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EXECUTIVE SUMMARY 



The Voluntary Public School Choice (VPSC) Program 
And Its Evaluation 

The No Child Left Behind Act of 2001 (NCLB) ^ expanded public school choice 

opportunities for students. First, the new accountability requirements in Title I of the 

Elementary and Secondary Education Act (ESEA) (Title I, Section 1 1 16(b)) required 

districts to offer public school choice to students in Title I schools that are identified for 

improvement, corrective action, or restructuring as a result of not meeting state definitions 

2 

for adequate yearly progress (U.S. Department of Education, 2004). 

Second, the law created the VPSC Program (Title V, Subpart 3, Section 5241 of the 
ESEA), which is the subject of the current report. The program supports the emergence and 
growth of choice initiatives across the country, by assisting states and local school districts 
in developing innovative strategies to expand choice options for students. 

The VPSC Program functions independently from the choice provisions in Title I and 
provides funds to a relatively small number of sites across the country. In October 2002, 
the U.S. Department of Education (the Department) awarded five-year competitive grants 
to 13 applicants. Awards ranged in size from $2.8 million to $17.8 million for an average 
award of $9.2 million for five years, or approximately $1.8 million per year. The VPSC- 
funded locations included: the state of Arkansas; Albany, N.Y.; Chicago, 111.; the state of 
Elorida; Hartsdale, N.Y.; Hillsborough County, Ela.; Ea Quinta, Calif; Miami, Ela.; the 
state of Minnesota; Swanzey, N.H.; New Haven, Conn.; Portland, Oreg.; and Rockford, 111. 

This report contains the final assessment of the first five years of the VPSC 
Program (2002-07). The evaluation was charged with assessing the VPSC Program’s 
progress in meeting the goals and fulfilling the intent of the original legislation. The 
evaluation addressed three central questions: 

1) What are the characteristics of the VPSC Program’s sites? 

2) How and to what extent does the VPSC Program promote educational 
equity and excellence? 

3) What academic achievement is associated with the VPSC Program? 



' The No Child Left Behind Act of 2001 (P.L. 107-1 10) amended the Elementary and Secondary 
Education Act of 1965. 

^ If all schools in a district have not made adequate yearly progress (AYP) for two or more consecutive 
years, the district must, “to the extent practicable,” establish a cooperative agreement for a transfer with other 
districts in the surrounding area. 34 C.F.R. 200.44(h)(1). 
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To address the third question, only a few of the VPSC sites provided the needed data. As a 
result, the evaluation’s findings for this third question may eome from too small a sample 
of sites or of VPSC ehoiee enrollees to eover the VPSC Program’s experienee adequately. 

In addition to the three eentral questions, the analysis also focused on four priorities in 
the original VPSC legislation: a) providing the widest possible choice to students in 
participating schools; b) promoting transfers of students from low- to higher-performing 
schools; c) forming interdistrict partnerships to allow students to transfer to a school in 
another district from that of their original school; and d) requiring sites to use funds to 
support transportation services for students (on the assumption that this would allow 
students to attend more attractive but also more distant schools). 

The evaluation covers choice initiatives at 13 sites that received VPSC grant awards 
from the U.S. Department of Education in the fall of 2002. The 2002 grantees were nine 
districts, three state education agencies, and one nonprofit organization.^ However, the 
findings of this report draw conclusions about the VPSC Program as a whole, not the 
activities at any given VPSC site. At the same time, the evaluation does not cover a second 
round of 14 five-year grants made by the VPSC Program in the summer of 2007. 

The relevant evaluation data came from several sources: multiple site visits to the 
VPSC sites; three rounds of surveys covering an average of 50 participating schools at each 
VPSC site; reviews of grantees’ annual reports; and the collection of archival data about 
student performance. The archival data tracked individual student achievement trends, 
covering one or more academic years prior to the start of a VPSC initiative and as many 
academic years as possible during the implementation of the initiative. The trends were 
then compared to those for a group of students not enrolled in a VPSC choice initiative. 



Brief Summary of Key Findings 

Briefly summarized, at the end of five years, the VPSC Program, though limited to 
grant awards at only 13 sites, showed a variety of different public school choice 
arrangements working in a diverse array of communities across the country. Among the 
four legislative priorities, 

• The VPSC Program made progress on the first priority in providing the 
widest variety of choice. 

• Transfers from low- to higher-performing schools comprised only a 
portion of the students enrolled in the VPSC initiatives, with only three 



^ The nonprofit organization operated charter schools and qualified as a school district, to be eligible for a 
VPSC award. 
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of the 13 sites limiting their enrollment to transfers from low- to higher- 
performing schools. 



• Most of the VPSC sites limited their choice initiatives to within-district 
options, rather than developing interdistrict options. 

• Relative to enrollment, transportation costs did not increase 
proportionately as might have been expected, because the VPSC 
initiatives permitted many students who were already attending distant 
schools to select schools closer to home. 

Regarding student achievement trends, data from a highly limited set of 
VPSC sites and students enrolling in their choice initiatives showed improved 
trends in math and reading, compared to matched groups of non-enrolling 
students. 

The gains were statistically significant, but the findings need to be tempered 
by several cautions: 

- the voluntary nature of choosing to enroll in a VPSC initiative and the 
possibility that the VPSC enrollees were more highly motivated students 
than non-enrollees, thereby accounting for some or all of the differences 
in student achievement; 

- the procedure for selecting the matched groups of non-enrolling 
students; 

- the length of time covered by the trends (more annual data points came 
from the years before rather than after enrollment); and 

- the fact that the data came from only four of the 13 VPSC sites and only 
six of 38 cohorts of annual enrollees across all of the VPSC sites. 

Given all the cautions, the findings in this analysis of six cohorts from four VPSC 
sites offer early promise regarding the potential benefits of the VPSC Program. 



Overview of the VPSC’s Sites 

The 13 VPSC sites were located across the country. Ten were located in 
predominantly urban communities, two in areas that cover both urban and rural regions, 
and one in an entirely rural area. 

Participating schools covered all grade levels: 64 percent spanned the elementary 
grades, 15 percent the middle school grades, 1 1 percent the high school grades, and 10 
percent were classified as “other” or had missing data. As a comparison, approximately 58 
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percent of schools nationwide are elementary schools, 17 percent are middle schools, 19 
percent are high schools, and 6 percent are “other.” 

The VPSC's choice initiatives varied in the timing of their implementation. Some of 
the sites started enrolling choice students as early as January 2003, while other sites started 
enrollment the following year or even later. Because all sites began some type of planning 
if not enrollment activity in 2002-03, the school year 2001-02 is considered the base year 
prior to the implementation of VPSC initiatives. The evaluation traced the sites’ activities 
for the duration of their five-year awards through 2006-07, although the majority of sites 
received no-cost extensions to support their operations for another year. 

Once they started enrollment, the VPSC sites offered choice options to a new set of 
students every year. Therefore, for each cohort of “first-time enrollees,” the “intervention 
period” varied. At the student level, this variation was taken into account by defining 
different base years for each cohort of students. A consequence of this staggered pattern is 
that more years of annual data were available for the older cohorts than for the younger 
ones. 



As with many other school districts in the country, nearly all of the 13 sites already 
had a broad variety of public school options prior to the VPSC Program. The program 
permitted sites to enhance or expand existing options, and sites did not need to start entirely 
new initiatives. Sites tried to make their entire array of choice options work well, but they 
did not necessarily track students’ participation on an option-by-option basis. 
Acknowledging this complication also is necessary for interpreting the findings in this 
report. For some of the sites, the VPSC funds were only part of the support for the 
identified initiative. As a result, the contribution of the VPSC Program may have been 
overestimated. At other sites, the VPSC funds not only supported the identified initiative 
but also partially supported other choice options at the same site. As a result, the 
contribution of the VPSC funds may have been underestimated. Unfortunately, the 
analysis was not able to distinguish the extent of these over- or underestimations. 

The VPSC Program also started in the same year that federal legislation expanded 
support for Title 1 choice options. The legislation allowed spending up to 20 percent of 
Title 1 funds to support transportation costs for students transferring away from schools 
designated under Title 1 as “identified for improvement.” Three of the 13 sites defined 
their VPSC initiatives to coincide or overlap closely with their Title 1 choice options. 
However, most VPSC sites defined schools participating in their VPSC initiatives as a 
broader set of schools than simply those designated under Title 1 as identified for 
improvement. 




Four Types of Choice Arrangements at the VPSC Sites 



The sites varied greatly in the design of their ehoice initiatives, differing in the 
number of students served, the number of partieipating publie sehools, and the eapaeity to 
aeeommodate transfers. In addition, the sites differed by how they defined ehoiee zones 
and managed the flow of students among participating schools. Despite this variation, sites 
also pursued some common paths. 

First, although unique, the VPSC’s choice initiatives tended to fall under four major 
categories, based on how sites defined choice arrangements and directed the flow of 
transferring students: 

• Five of the sites designated specific schools to be either sending schools 
or receiving schools but not both. 

• Five sites defined initiatives whereby the same schools could be both 
sending and receiving. 

• One site established a within-school initiative, in which students chose 
from education programs within the same school and did not transfer 
between schools. 

• Two sites had initiatives involving a mixture of the first three types. 

Second, all sites focused on two core activities throughout the implementation 
process: 1) engaging parents and community members, and 2) building capacity at schools 
to attract and accommodate choice transfers. 

Parent and community activities included a rich array of outreach, marketing, and 
communication efforts. Sites also engaged parents and community representatives in 
developing and implementing choice initiatives. 

Capacity-enhancing activities included starting new academic programs or subjects, 
purchasing supplies and equipment for schools, and providing professional development to 
teaching staff. However, the sites’ capacity-enhancing activities were not necessarily 
accompanied by an expansion of seats or classrooms at many of the receiving schools. For 
instance, none of the sites reported hiring more teaching staff or taking other steps simply 
to expand the number of seats to accommodate higher enrollments at the existing schools. 



Student Participation at the VPSC Sites 

In 2005-06, across 12 of the 13 VPSC sites, 24,921 students enrolled in choice 
initiatives, reflecting an overall participation rate of 2.8 percent of the students eligible to 
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enroll. This number of students represented those who had enrolled for the first time that 
year (“first-time enrollees”). The eumulative first-time enrollees, starting from the 
ineeption of the VPSC Program in 2002-03 until 2005-06, ineluded 49,616 students. 

Thus, the most aeeurate estimate of partieipation falls somewhere between 24,921 and 
49,616. (However, the eumulative enrollment overestimates the total enrollment beeause 
of an attrition faetor that was not traeked by the sites.) 

Overall, the number of enrolling students in the VPSC Program increased during 
the earlier years of the program but declined in the program’s fifth year. Of the VPSC 
sites’ enrolling students, ten provided eligibility and enrollment data for four consecutive 
years (2003-04 to 2006-07). These data permitted the estimation of trends through the 
fifth VPSC year (2006-07). The yearly data captured the number of first-time (i.e., new) 
enrollees each year, not the total number across all years. For these “first-time enrollees,” 
the trends showed that the VPSC Program at first averaged 696 enrollees per site in 2003- 
04, then reached a peak of 2,459 per site in 2005-06, and then declined to 2,167 per site in 
2006-07. 

The decline may have reflected the actual saturation of the VPSC initiatives because a 
good (but unmeasured) portion of the earlier years’ enrollees still remained enrolled in the 
later years, possibly limiting the seats available for first-time enrollees in the final year. 

Participation rates also showed the same pattern, increasing from 2003-04 to 2005- 
06, and then declining in 2006-07. For the same ten sites reporting data from 2003-04 to 
2006-07, the participation rates by first-time enrollees also increased from 1.5 percent to 
4.1 percent, from 2003-04 to 2005-06, and then dropped to 3.0 percent in 2006-07. 



Progress on Program Priorities 

The VPSC legislation had four program priorities. Three priorities were: to provide 
the widest variety of choices; to encourage transfers from low- to higher-performing 
schools; and to provide opportunities for students to transfer to schools outside of their 
home districts. The fourth priority directed sites to use some of their VPSC funds to 
support transportation services and costs. Findings on these four priorities are discussed 
next. 



The VPSC Program made progress on the first priority of providing the widest 
variety of choice. Sites expanded the assortment of choice options in participating schools 
and offered a diverse number of academic programs to transferring students. The VPSC 
initiatives also made efforts, through media campaigns and related activities, to increase 
parents’ awareness of the variety of education options available to them. 

Transfers from low- to higher-performing schools comprised only a portion of the 
students enrolled in VPSC initiatives. Only three VPSC sites limited their enrollment to 
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transfers from low- to higher-performing sehools. Another five sites permitted transfers 
within an entire distriet or zone, but only two of the sites tracked the portion of the transfers 
from low- to higher-performing schools. Aside from the five sites that either limited or 
tracked their transfers from low- to higher-performing schools, none of the other eight sites 
could provide such information. At the five sites, the confirmed transfers from low- to 
higher-performing schools represented 1,295 of 5,927, or 21.8 percent, of their total 
transfers from 2005-06. 

The other eight VPSC sites either permitted a wider variety of transfers or had VPSC 
enrollments involving no transfers. Among the sites not tracking the low- to higher- 
performing transfers, one received a waiver from the Department to omit such tracking 
because all of the site’s enrollees were low-performing students (scoring “below proficient” 
on the state assessment), but they were not necessarily transferring from low- to higher- 
performing schools. Thus, across the entire VPSC Program, the actual portion of low- to 
higher-performing transfers could be larger or smaller, depending on the number of 
transfers at the eight sites that did not track or document the pattern of their transfers. 

Most of the VPSC sites limited their choice initiatives to within-district options, 
rather than developing interdistrict partnerships. Although the implementation of formal 
interdistrict choice options might have expanded the variety of choices available to students 
even further, only five of the 13 VPSC sites created formal interdistrict options. At the 
eight remaining sites, many had existing transfer agreements with neighboring districts. 
However, these transfer options were separate from VPSC initiatives and were generally 
reviewed on a case-by-case basis. The majority of the VPSC initiatives did not directly 
promote student transfers to other school districts, and VPSC funds were not used to 
support interdistrict transfers at these sites. 

Relative to enrollment, transportation costs did not increase proportionately as 
might have been expected. This is partly because the VPSC initiatives permitted many 
students who were already attending distant schools to select schools closer to home. As an 
example, two of the VPSC sites had recently emerged from court-ordered school busing, 
from which students had been assigned to more distant schools as part of the original 
desegregation order. The VPSC-funded initiative gave affected students the choice of 
enrolling at neighborhood schools. Under these circumstances, sites experienced minimal 
or reduced transportation costs. 



Findings on Student Achievement Trends 

The final student achievement trends came from six cohorts of VPSC enrollees and 
six corresponding groups of non- VPSC enrollees, across four of the 13 VPSC sites. The 
cross-site analysis first estimated the student achievement trends for these two groups 
separately and then compared the trends between the two groups. 
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These two analyses served two different purposes. The first, estimating trends 
separately, was needed to establish whether either the enrollee or matched comparison 
group was alone moving in a positive or negative direction. The goal was to determine 
whether the enrollee group might have been performing worse, regardless of any relative 
difference between it and the comparison students. The second, estimating trends relative 
to each other, then captured the comparison between the two groups. (The two analyses 
involved two different units of analysis, students and cohorts, and the values should not be 
compared across these two analyses.) 

When the VPSC and non-VPSC trends were examined separately, the VPSC 
enrollees’ trends were neutral in math and positive in reading, but not statistically 
significant. More important, the enrollee group’s scores were not found to be declining in 
any way. In contrast, the non-enrollee group showed a declining trend in math proficiency 
that was statistically significant at the 99 percent confidence level. The non-enrollees’ 
trends for reading were positive but not significant. 

When the VPSC and non-VPSC trends were compared, students enrolling in the 
VPSC initiatives had better student achievement trends than those not enrolling. The 
enrollees surpassed the non-enrollees in the trends for both mathematics and reading, with 
the differences in mathematics being statistically significant at the 99.9 percent confidence 
level and in reading at the 99 percent level. The comparison was based on a meta-analysis 
of effect sizes across all six of the cohorts, with the individual effect sizes for each cohort 
having already accounted for the demographic and baseline differences between the VPSC 
and non-VPSC groups. Because the units of measure were standardized scores, the trends 
cannot easily be translated into everyday educational units, but the effect sizes (.020 and 
.028 for mathematics and reading respectively) appear to be modest. 

These findings need to be accompanied by several cautions about interpreting the 
differences in achievement trends between the enrolling and non-enrolling groups. The 
main cautions are as follows. 

First, in all of the choice initiatives, students can choose to enroll or not. As a result, 
an enrollee group may represent more highly motivated students or differ in other 
unobserved ways from a non-enrollee group, accounting for some or all of any subsequent 
differences in student achievement.^ 



* Confidence levels represent the probability that observed differences do not differ by chance alone but 
represent actual differences between two groups. Statistically, the analyses were based on the use of random- 
effects models, which maximize the appropriate use of all of the sites’ data submitted for each of the six 
cohorts. Unlike fixed-effects models, the random-effects models assume that the available data come from 
but a sample of a fuller universe of students and that the sample therefore is likely to contain additional 
variance due to sampling errors (Bryk and Raudenbush, 1992). 

^ To overcome self-selection and other biases, one frequently raised possibility has been to use data from 
sites’ lotteries as if they were “natural experiments” (sites often use lotteries to select participants when the 
number of students exceeds the available choice opportunities). However, the lottery losers may not actually 
serve as the needed counterfactuals for lottery winners (see fuller discussion in appendix A). 
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Second, the trends in the present analysis leaned in the direction of having more of the 
data points precede rather than follow enrollment in the VPSC initiative. Confidence about 
the trends concurrent with a VPSC initiative, much less inferences about any effects, would 
be increased if analyses were based on a larger number of data points following enrollment. 

Third, the data only came from a small sample of sites (four of 13) and from only a 
small sample of the enrolling students (six of 38 cohorts of annual enrollees across all of 
the VPSC sites by 2005-06). Data from more sites and cohorts would produce a firmer set 
of findings about the program as a whole. The ongoing VPSC Program can put renewed 
emphasis on obtaining such data, given the new round of five-year awards made in 2007. 
The possibilities of such additional data are especially strong, given that seven of the newly 
awarded sites were continuations from the first round of awards. 

Given the cautions, the findings in this analysis of six cohorts from four VPSC sites 
offer early promise regarding the potential benefits of the VPSC Program. 

Recommendations for improving future evaluations as the VPSC Program progresses 
with its 2007 cohort of grantees are detailed at the end of the full report. Aside from taking 
steps to overcome the earlier cautions about interpreting student achievement findings, two 
specific procedures could improve the robustness of future analyses: 

• Careful recordkeeping and tracking of all participants. Sites should 
keep accurate and aggregate counts of all three types of choice 
participants (eligibles, applicants, and enrollees) and also should 
compile individual student-level data for choice enrollees’ 
demographic characteristics and student achievement scores. 

• A modified procedure for defining comparison students. Sites should 
not try to match any particular group of non-enrolling students, as the 
sites did in the current evaluation. Rather, the more desirable 
procedure would be for sites to provide data from a larger but 
nonselected set of non-enrolling students, such as those students 
remaining in the sending schools or even a districtwide set of 
students. Analytic procedures such as propensity score matching or 
some similar procedure conducted by the evaluation team could then 
provide a fairer selection of matched comparison group than the sites’ 
procedures in the current evaluation. 
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1. THE VOLUNTARY PUBLIC SCHOOL CHOICE (VPSC) PROGRAM 

AND ITS EVALUATION 



The No Child Left Behind Act of 2001 (NCLB)^ expanded publie sehool choice 
opportunities for students. First, the new accountability requirements in Title I of the 
Elementary and Secondary Education Act (ESEA) (Title I, Section 1 116(b)) required 
districts to offer public school choice to students in Title I schools that are identified for 
improvement, corrective action, or restructuring as a result of not meeting state definitions 
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for adequate yearly progress (U.S. Department of Education, 2004). 

Second, and the subject of the present report. Congress created the VPSC Program 
(Title V, Subpart 3, Section 5241 of the ESEA) to support the emergence and growth of 
choice initiatives across the country. The purpose of the program is to assist state and local 
school districts in the development of innovative strategies to expand options for students, 
and to encourage transfers of students from low- to higher-performing schools. 

The VPSC Program functions independently from the choice provisions in Title I and 
provides funds to a relatively small number of sites across the country. The program has 
made two rounds of awards, only the first of which is the subject of the present evaluation. 
The first round occurred in the fall of 2002, when the U.S. Department of Education (the 
Department) awarded five-year grants to 13 applicants (see exhibit 1-1). The second round 
occurred in the summer of 2007, when the Department made another set of five-year grants 
to 14 applicants, seven of which had received awards in the first round. 

The VPSC Program was designed and initiated against a backdrop of increasing 
interest in public school choice. Such initiatives give students the option of enrolling at a 
public school other than the one to which they are assigned as a result of their residential 
location. Their choices can include: all the other public schools in a system; some of the 
schools (e.g., schools in a geographic region within the system); or specific schools (e.g., 
magnet schools, charter schools, or schools identified under Title I choice). Despite the 
variation in options, all choice is limited to public schools only. 

The evaluation of the VPSC Program, focusing on the first round of awards only, 
addressed three questions to assess the VPSC Program’s progress in meeting the goals and 
fulfilling the intent of the original legislation. The three central questions Congress asked were: 



^ The No Child Left Behind Act of 2001 (P.L. 107-110) amended the Elementary and Secondary 
Education Act of 1965. 

’ If all schools in a district have not made adequate yearly progress (AYP) for two or more consecutive 
years, the district must, “to the extent practicable,” establish a cooperative agreement for a transfer with other 
districts in the surrounding area. 34 C.F.R. 200.44(h)(1). 
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Exhibit 1-1 



Location of the VPSC Grantees 
(2002 Awards) 



state of Minnesota 




Rockford Public School 




Chicago Public 




Monadnock 


Department of 




District #205, III. 




Schools, III. 




Regional School 


Education 




Award: $10,095,054 




Award: $10,225,138 




District, N.H. 


Award: $1 1 ,751 ,052 


1 "'*'*'**Niw \ 


/ r\ 




Award: $8,396,542 



Portland Public 
Schools, Oreg. 

Award: $6,467,122 



Desert Sands 
Unified School 
District, Calif. 

Award: $7,947,309 




Greenburgh 
Central School 
District No. 7, N. Y. 

Award: $2,822,155 



Brighter Choice 
Charter Schools, N. Y. 

Award: $3,41 1,567 



Miami-Dade 
County Public 
Schools, Fla. 

Award: $11,717,901 



School District of 
Hillsborough County, Fla. 

Award: $10,226,759 



Exhibit Reads: One of the 13 VPSC grantees, Portiand Pubiic Schoois in Oregon, received a five- 
year VPSC award of $6,467,122. 



1) What are the charaeteristies of the VPSC Program’s sites? 

2) How and to what extent does the VPSC Program promote edueational 
equity and exeellence? 

3) What aeademie achievement is associated with the VPSC Program? 

Question one relates to basic descriptive information about the program sites and their 
implementation strategies, including activities related to community outreach and capacity- 
building within participating schools. 
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Question two relates to the extent and nature of student partieipation in the ehoiee 
initiatives funded by the program, as well as sites’ pursuit of the stated goals of the VPSC 
legislation. The legislation stipulated four priorities in the seleetion of sites: 

a) Provide the widest possible ehoiee to students in partieipating sehools; 

b) Promote transfers of students from low-performing to higher-performing 
sehools; 

c) Inelude interdistrict partnerships to allow students to transfer to a school 
in another district from that of their original school; and 

d) Require sites to use funds to support transportation services for students 
(on the assumption that this would allow students to attend more 
attractive but also more distant schools). 

Question three pertains to Congress’s interest in having the evaluation investigate the 
achievement outcomes associated with the VPSC Program. It asks whether students who 
participated in the VPSC Program had greater academic outcomes in reading and math 
compared to similar students that had not participated in the program. 

This report covers the three evaluation questions and related topics. After a brief 
description of the methodology in chapter 2, the report is organized as follows: 

Chapter 3 addresses the first evaluation question by providing a detailed analysis of 
the characteristics of the VPSC Program’s grantees. This chapter also categorizes the sites 
by type of choice arrangement, documents important program practices, and describes 
other choice initiatives at the VPSC sites. 

Chapter 4 addresses the second evaluation question and discusses how and to what 
extent the program has promoted educational equity and excellence. The chapter presents 
trends in eligibility, applicants, and enrollees at the sites. This chapter also examines the 
degree to which the program has provided students with: a wide variety of choice; 
opportunities to transfer from low- to higher-performing schools; interdistrict choice 
options; and transportation support. 

Chapter 5 addresses the third evaluation question by presenting and discussing 
student achievement trends concurrent with the VPSC Program. 

Finally, Chapter 6 of the report identifies useful choice practices emerging from the 
VPSC Program and discusses implications for the program and future research. 
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2. SUMMARY OF EVALUATION METHODOLOGY 



The purpose of the national evaluation was to assess the experienee of the VPSC 
Program. The evaluation followed a mixed quantitative and qualitative methods researeh 
design, with data eoming from a variety of original and arehival sourees (for a full 
description of the evaluation methodology, see appendix A: Detailed Evaluation 
Methodology). 



2.1 Evaluation Design for a Program Evaiuation 

The evaluation covered the 13 sites that received VPSC awards from the Department 
in the fall of 2002. However, the analysis and findings in this report aimed at drawing 
conclusions about the VPSC Program as a whole, not the activities at any given VPSC 
Site. 



Because choice initiatives can involve schools and students in a variety of roles, the 
evaluation defined the schools’ roles in the following manner: 

(a) Sending schools: schools from which students transfer; 

(b) Receiving schools: schools to which students transfer; 

(c) Both sending and receiving: schools that have students both 
transferring in and out as part of the same choice initiative; and 

(d) Within-school initiatives: initiatives with no inter-school transfers, 
because students choose among different education programs within 
the same school. 

Similarly, the evaluation defined the students’ roles in the following manner: 

(a) Eligible students: all students who could potentially participate in a 
VPSC initiative; 

(b) Applicants: the set of eligible students who applied either to attend 
another public school or to participate in an academic program within 
their original school; and 

(c) Enrollees: those students who successfully applied and enrolled in a 
school or program as a result of a VPSC initiative. 



* Each site engaged a “local” evaluator to assess the specific site’s progress and accomplishments. 
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The evaluation aecounted for the varied timing of the VPSC’s choice initiatives in the 
following manner. After receiving their five-year awards in the fall of 2002, some of the 
sites started enrolling choice students as early as January 2003. Other sites carried out 
planning activities first and then started enrollments the following year or later. Because 
all sites began some type of activity in 2002-03, the school year 2001-02 is considered the 
base year prior to the implementation of the VPSC initiatives. The evaluation traced the 
sites’ activities for the duration of their five-year awards, through 2006-07. 

Sites also offered choice options to a new set of students every year. Therefore, for 
each cohort of students, the “intervention period” varied. At the student level, this 
variation was taken into account by defining different base years for each cohort of 
students. A consequence of this staggered pattern is that more years of annual data were 
available for the older cohorts than for the younger ones. 

Finally, the evaluation design called for the collection of data from comparison 
groups for every VPSC site. The groups mainly consisted of students who were not 
enrolled in the VPSC initiatives but who had similar demographic and academic 
characteristics as the enrolling students. The VPSC sites provided data about these 
comparison groups. (See chapter 5 for additional details.) 

From a design standpoint, two complications diminished the evaluation’s ability to 
assess the VPSC Program. First, all of the sites already had one or more choice options 
ongoing at the onset of the VPSC initiative. These other options included magnet and 
charter schools. Title I public school choice, and a variety of other choice options offered 
by the district or state. Overall, even when sites used VPSC funds exclusively to support a 
new VPSC Program initiative, the sites had other unrelated choice options operating at the 
same time (see exhibit 2-1). 

Second, the VPSC Program permitted sites to expand existing arrangements rather 
than to establish new ones. The majority of the sites chose to expand existing initiatives. 
Separating the VPSC-funded portion of a choice initiative from other aspects of the 
initiative has not been easily delineated. 



2.2 Data Collection 

The data used in this evaluation came from a variety of sources. These included: 
multiple site visits to the VPSC sites; surveys covering an average of 50 participating 
schools at each VPSC site; and the collection of archival data about student achievement as 
well as reviews of the grantees’ reports to the Department. 

The team conducted three rounds of site visits to every grantee site. The site visits 
covered the VPSC project site as well as one sending school and one receiving school (or 
two participating schools at those sites not designating specific sending and receiving 
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Exhibit 2-1 



Other Public School Choice Options at the VPSC Sites 



Other choice options 


Number of sites with other public 
school choice options in addition 
to the VPSC-funded initiative* 


Magnet schools 


13 


Charter schools 


11 


Title I choice 


11 


Interdistrict options 


9 


Other district options 


9 


Other state options 


5 


Total for all 13 sites 


58 


Average for each site 


4 or more** 



Exhibit Reads: Aii 1 3 sites had magnet schoois in addition to the VPSC-funded initiative. 
*Individual VPSC sites can appear under more than one type of choice option. 

**Each of the 13 sites had an average of four or more public school choice options in addition to the VPSC- 
funded initiative. 

Sources: Analysis of site visit data and Grant Performance Reports, by COSMOS Corporation, 2007. 



schools). The sites ehose two schools that had a lot of choice aetivity (i.e., many 
transferring students), so that the site visit team could observe the VPSC initiative in action 
at the school level. 

A two-person team eondueted all three rounds of field visits. The team’s data 
collection activities were guided by field protocols (see appendix B). These protocols 
included interviews with the VPSC site’s projeet director, staff, and other key participants 
in the choice initiative. The protocols also called for the colleetion of doeuments and 
arehival data related to the interviews. 

The team also conducted three rounds of surveys with participating schools at VPSC 
sites. The surveys were brief, mainly gathering data to eorroborate the sehools’ 

9 

participation in initiatives but not to investigate the schools’ other conditions in any depth. 



^ The survey eovered 12 of the 13 sites in the VPSC Program beeause the 13th site foeused mainly on 
teehnieal assistanee to distriets statewide. 
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The surveys were based on a elosed-ended questionnaire (see appendix B) sent to 
school principals. The questionnaire requested information about: student demographics; 
school performance; the choice options available to students; the percentages of students 
taking part in these options; the methods by which choice information was being shared 
with parents; and whether staff members were receiving professional development related 
to school choice. 

The survey included all of the schools that were eligible to participate in the choice 
initiative. One statewide initiative was an exception: no schools were surveyed in this 
initiative; instead, the districts with five or more students participating in the choice initiative 
were asked to complete the survey at the district level. 

In the survey’s third and final round, the evaluation team distributed questionnaires to 
689 schools. Respondents at 630 schools completed and returned the questionnaire, 
resulting in an overall response rate of 91 percent. The response rates for the earlier two 
rounds were 75 and 91 percent, respectively (see appendix A, exhibit A-2). 

The data collection also included archival data. The data covered individual student 
achievement scores for one or more academic years prior to the start of a VPSC initiative 
through as many academic years as possible during the implementation of the initiative. 
The data covered both the students enrolled in the VPSC choice initiatives and matched 
comparison groups of students. 

The archival data also came from reviews of annual performance reports (2003-06) 
submitted by the VPSC grantees to the VPSC Program Office at the Department. Each 
year, the office issues reporting requirements, based in part on the data collection 
suggestions by the national evaluation team. However, the review did not include the 
grantees’ final performance reports, which were to be submitted after the conclusion of the 
national evaluation. 



2.3 Analytic Priorities 

2.3.1 Cross-site Analysis 

The general analytic strategy in dealing with evaluation data was to examine the 
extent of any relationship between a VPSC initiative and concurrent educational outcomes, 
including equity, excellence, and student achievement. 

The analysis began by focusing on within-site conditions. A separate database, 
consisting of narrative and numeric material, brought together all of the data about a single 



*** At this site, the distriet was the appropriate respondent because many of the participating students were 
not assigned to specific schools, either prior to or as part of the VPSC initiative. 
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VPSC initiative and addressed sueh issues as: (1) distinguishing ongoing ehoiee options 
from the VPSC initiative at each site; and (2) calling attention to contextual conditions 
potentially related to the VPSC initiative. The creation of this database was itself a 
qualitative task and aimed at strengthening the possible association between VPSC- 
supported actions at a site and later outcomes. An important part of this procedure was to 
search for and understand the role of possible rival explanations (Yin, 2000). For instance, 
ongoing district or school policies apart from the VPSC initiative may be strongly 
associated with the observed outcomes. 

The primary focus of the national evaluation was then to conduct a cross-site analysis 
to arrive at findings for the VPSC Program as a whole, rather than to assess the 
accomplishments at any given site. The cross-site analysis identified patterns across the 
VPSC sites. For instance, findings and lessons from the VPSC Program revealed different 
types of choice initiatives, with the 13 sites arrayed into subgroups based on the design of 
their VPSC-funded choice arrangement (see chapter 3). The subgroups helped provide 
greater insight into the associated outcomes from choice initiatives and also served as 
practical examples to be considered for implementation by other districts in the future. 

The practical examples were part of an effort that came from an expressed need, by 
the Department, for the national evaluation to report new information about useful practices 
for conducting choice initiatives. From the national evaluation, such information on choice 
initiatives would serve at least two audiences: 1) districts implementing the choice 
provisions of Title I, and 2) districts wanting to start or strengthen their own public school 
choice initiatives independent of the Title I requirements. 

2.3.2 Student-level Analysis 

The evaluation collected and analyzed student achievement trends at the VPSC sites. 
The evaluation invested extensive efforts in getting sites to collect the needed student 
achievement data. Data cleaning and clarification of the data files submitted by the sites 
also consumed time and resources. Chapter 5 describes the methodological techniques and 
the findings (also see section 3, “Methods for Analyzing Student Achievement Data” in 
appendix A for the detailed analysis plan). 
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3. CHARACTERISTICS OF THE VPSC PROGRAM’S SITES 



The Department awarded grants to three types of organizations in the fall of 2002; 

1) nine loeal or regional school districts; 2) three state education agencies; and 3) one 
nonprofit, charter school organization. ' ' Each type of organization engaged in a unique 
choice initiative. Despite this variation, sites pursued some common paths. First, the 
school choice initiatives tended to fall under four major categories based on how sites 
defined choice arrangements and directed the flow of enrolling students. Second, all sites 
focused on two core activities throughout the implementation process: 1) engaging parents 
and community members; and 2) building capacity at schools to attract and accommodate 
choice enrollment. 



3.1 Overview of VPSC Sites 

3.1.1 Sites’ Characteristics 

The 13 VPSC sites were located across the country. Ten were located in 
predominantly urban communities, two in areas that cover both urban and rural regions, 
and one in an entirely rural area (see exhibit 3-1). Ten of the locales represented a total 
population of over 100,000 people each. The public school student populations were 
mostly diverse and of low-income backgrounds. Nonwhite students comprised over 60 
percent of the student population at seven school systems represented by the sites. 
Similarly, over 60 percent of the students were eligible for the Free and Reduced-Price 
Funch Program at seven sites. 

3.1.2 Brief Descriptions of Sites’ Choice Initiatives 

The VPSC sites varied greatly in the design of their choice initiatives. The sites 
differed widely in the number of students served, the number of participating public 
schools, and the capacity to accommodate transfers (see exhibit 3-2 for brief descriptions). 
They differed by how they defined choice zones and managed the flow of students among 
participating schools. Furthermore, four sites used VPSC funds to support a preexisting 



*’ The nonprofit organization operated charter schools and qualified as a school district, to be eligible for 
a VPSC award. 

The Free and Reduced-Price Lunch Program is part of the National School Lunch Program 
administered by the U.S. Department of Agriculture. The program provides nutritionally balanced, low-cost 
or free lunches to more than 26 million students each school day. Children tfom families with incomes at or 
below 185 percent of the poverty level are eligible for free and reduced-price meals. For the period July 1, 
2004, through June 30, 2005, 185 percent of the poverty level was $34,873 for a family of four (USDA, 
2004). 
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Exhibit 3-1 



Community Characteristics 



Characteristic 


No. of sites 


Type of community: 




Urban 


10 


Rural 


1 


Mixed 


2 


Population: 




Under 100,000 


3 


100,000 to 1 million 


6 


Over 1 million 


4 


Public school enrollment: 
No. of students enrolled 




Under 25,000 


4 


25,000-100,000 


4 


Over 100,000 


5 


Percent nonwhite 




Under 30 


1 


30-60 


5 


Over 60 


7 


Percent eligible for the Free and 
Reduced-Price Uunch Program 




Under 30 


1 


30-60 


5 


Over 60 


7 



Exhibit Reads: Ten of the VPSC sites are iocated in urban areas. 
Source: NCES, Common Core of Data, 2004-05. 



or preplanned expansion of districtwide choice. Nearly all of the others used the funds to 
support a specific function within an environment that already had other existing choice 
options. Only two sites had minimal choice options prior to VPSC because they had no 
Title I schools and only a preexisting magnet program. 
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Exhibit 3-2 



Brief Descriptions of the VPSC Sites’ Choice Initiatives 



VPSC grantee 


Award 

Amt. 

($m) 


Description of initiative, from 2002-03 to 2005-06 


A. PREEXISTING OR PREPLANNED DISTRICTWIDE INITIATIVES 


Chicago Public 
Schools, III. 


$10.2 


Has augmented preexisting districtwide choice by using VPSC funds to support schoois in 
neighborhood iearning ciusters (NLCs) of four-to-six K-8 schoois each. Four ciusters 
started in 2003-04, three started in 2004-05, and three started in 2005-06. There is a new 
schooi in each ciuster, and existing schoois deveiop magnet themes; ciusters have 
coordinators; and schoois receive VPSC funds. 


School District 
of Hillsborough 
County, Fla. 


$10.2 


Has supported two cohorts of students (K-12) starting in 2004-05, enroiling in a districtwide 
controiied choice initiative, invoiving seven regions (and an urban “zone” within each region) 
and the creation or expansion of “attractor” programs at existing schoois to maintain or 
increase student diversity. Pians for the initiative were in piace weii before VPSC, which 
oniy partiaiiy supports the initiative. 


Minnesota 
Department of 
Education 


$11.8 


Has supported three cohorts of transfer students in a preexisting program aiiowing MPS 
students quaiifying for the Free and Reduced-Price Lunch Program to transfer to neariy 60 
schoois in eight surrounding suburban districts, and suburban or MPS students to attend 12 
schoois in MPS (K-12). VPSC funds partiaiiy support Parent information Centers, some 
transportation, and reiated support services. 


Portland Public 
Schools, Oreg. 


$6.5 


Has supported three cohorts of transfer students through the augmentation of a districtwide, 
previousiy avaiiabie choice program (K-12). VPSC funds heip support new enroiiment and 
transfer poiicies; seiection, iottery, and transition services; and coiiaborative curricuia 
pianning. 


B. INITIATIVES ENHANCING OTHER EXISTING OPTIONS 


Brighter 
Choice Charter 
Schools, N.Y. 


$3.4 


Has supported the first cohort of students by opening three new charter middie schoois in 
Aibany in 2005-06. The site has continued to support charter schooi deveiopment and to 
coordinate suppiementai educationai services. 


Desert Sands 
Unified School 
District, Calif. 


$7.9 


Has supported four cohorts of students transferring from iow-performing sending schoois to 
higher-performing receiving schoois (K-12). Funds heip augment curricuia at receiving 
schoois with environmentai studies theme to make them more attractive to transferring 
students. 


Florida 

Department of 
Education 


$17.8 


Has provided technicai assistance to schooi districts as they deveiop and impiement their 
choice pians. Funds assist a subset of mentor districts (aiready successfui at choice 
options) and mentee districts (needing to expand options), and postsecondary institutions 
(to support schooi choice information and technicai assistance centers). 


Miami-Dade 
County Public 
Schools, Fla. 


$11.7 


Has supported three cohorts of students (K-12) in two of eight transportation zones in the 
district. Provides funds to create choice programs at under-enroiied schoois. The site has 
created or enhanced choice programs, starting in 2003-04 with one new “commuter” schooi 
in Zone 1 ; in 2004-05, expanded to a totai of nine schoois in Zone 1 , and four schoois in 
Zone 2; and in 2005-06 added one additionai choice program in each of the two zones. 


New Haven 
Public Schools, 
Conn. 


$9.5 


Has supported four cohorts of students transferring from iow-performing schoois to identified 
higher-performing schoois, inciuding Lighthouse Schoois, magnets, charters, and suburban 
pubiic schoois. VPSC funds expanded programs at four Lighthouse Schoois (K-6) in 2002- 
03; five Lighthouse Schoois in 2003-04; and three in 2004-05 and 2005-06. 


Rockford 
Public School 
District #205, 

Bl. 


$10.1 


Has supported four cohorts of students transferring from iow-performing schoois (K-8) to 
identified receiving schoois. VPSC funds provide support to receiving schoois; the Parent 
Resource Center; parent and transportation services; and tutoring programs run by 
community or faith-based organizations. 



(Exhibit 3-2, Section C begins on the next page) 
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(Exhibit 3-2, continued) 



VPSC grantee 


Award 

Amt. 

($m) 


Description of initiative, from 2002-03 to 2005-06 


C. INITIATIVES CREATING A NEW CHOICE OPTION 


Arkansas 
Department of 
Education 


$9.3 


Has supported four cohorts of students attending an off-campus (residential or 

community) program to receive a rigorous and comprehensive, self-paced education, 
delivered online and aligned with the state’s standards (requires students to take 
state assessment). The program covered K-5 in 2002-03, K-7 in 2003-04, and K-8 
starting in 2004-05. 


Greenburgh 
Central School 
District No. 7, 

N.Y. 


$2.8 


Has supported three cohorts of middle and high school students (7-12) attending new 
academies of choice within middle and high school. Has also supported two cohorts 
of elementary students having expanded to the elementary level in 2004-05 with the 
implementation of the International Baccalaureate Primary Years Programme (IB) to 
offer enhanced programming to all K-6 students. 


Monadnock 
Regional School 
District, N.H. 


$8.4 


Has supported four cohorts of students making interdistrict transfers (9-12), attending 
new programs in their original schools (6-12), transferring to an alternative high 
school (MC2) and a virtual high school, or enrolling in college courses. In 2005-06, 
expanded interdistrict choice to the elementary level and has added new choice 
programs at two high schools and two middle schools. 



Exhibit Reads: Chicago Pubiic Schoois received an award of $10.2 miiiion to augment a preexisting 
districtwide choice program. 

Sources: Analysis of site visit data and Grant Performance Reports, by COSMOS Corporation, 2007. 



3.1.3 Participating Schools 

The majority of participating schools were elementary schools. Participating 
schools covered all grade levels: 64 percent spanned the elementary grades, 15 percent the 
middle school grades, 1 1 percent the high school grades, and 10 percent were classified as 
“other” or had missing data (see exhibit 3-3). As a comparison, approximately 58 percent 
of schools nationwide are elementary schools, 17 percent are middle schools, 19 percent 
are high schools, and 6 percent are “other.” Although most of the sites focused their 
choice initiatives broadly across all grade levels, four VPSC sites only targeted grades 
K-8, which may account for the slightly higher prevalence of participating elementary 
schools. 



3.2 Four Types of Choice Arrangements at the VPSC Sites 

The VPSC Program allowed grantees to design choice initiatives to meet their own 
needs. Nevertheless, going beyond the unique circumstances at each site, the initiatives 
fell into a four-fold typology of choice arrangements (see exhibit 3-4). 
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Exhibit 3-3 



Grade Span Distribution of Participating Schools* 



School level 


Number Percent 


Elementary school 
Middle school 
High school 
Other** 

Not applicable/missing*** 


439 63.7 

105 15.2 

73 10.6 

19 2.8 

53 7.7 


TOTAL 


689 100.0 



Exhibit Reads: 439 (or 63.7 percent) of the schoois were eiementary schoois. 

*Includes 12 of the 13 sites in the VPSC Program because the 13th site focused mainly on technical 
assistance to districts. 

**“Other” is defined by NCES as any grade span configuration, including ungraded, not falling within the 
three categories of elementary, middle, and high. 

***“Not applicable/missing” contains those that either did not have school-level CCD data available or did 
not have data available for the grade span variable. 

Source: NCES, Common Core of Data, 2004-05. 



Exhibit 3-4 

Four Types of Choice Arrangements 



Type of choice arrangement 


No. of VPSC 
sites 


No. of schools* 


Predesignated sending or receiving schools 


5 


95 


Same schools are both sending and receiving schools 


5 


524 


Within-school options only 


1 


5 


Mixture of the first three groups 


2 


6 


TOTAL 


13 


630 



Exhibit Reads: Five VPSC sites impiemented choice arrangements with predesignated sending or 
receiving schools. 

*Based on the 630 School Survey respondents. The survey covered 12 of the 13 sites in the VPSC Program 
because the thirteenth site focused mainly on technical assistance. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 
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First, five initiatives designated specific schools to be either sending schools or 
receiving schools but not both. In this first type, students attending predesignated sending 
schools were eligible to transfer, and their choices were limited to a predesignated group of 
receiving schools. At these sites, VPSC funds mainly supported the strengthening or 
capacity-enhancement at the receiving schools. 

Under this first arrangement, three of the five sites defined their sending schools as 
“low-performing” according to the NCLB criteria regarding schools identified for 
improvement. However, because the VPSC legislation did not prescribe a standard for 
identifying “higher-performing” schools, the program and its sites defined the receiving 
schools simply as those that were not “low-performing” schools. During the second year 
of its VPSC initiative at one of these sites, initially eligible receiving schools later became 
identified for improvement. At that point, the site decided these schools would be 
ineligible to serve as receiving schools. 

The fourth and fifth sites in this first type of arrangement predesignated sending and 
receiving schools but did not attempt to limit either group according to any low- to high- 
performing criteria. One of these sites started several new receiving (charter) schools 
whose performance could not, by definition, be known at the time when transfers began. 
The last of these sites defined all schools in the system as possible sending schools, and the 
receiving “schools” were predesignated off-campus sites. 

Second, five initiatives defined the same schools as both “sending” and 
“receiving. ” Under a second type of arrangement, five sites permitted transfers between 
all public schools, either districtwide or within pre-specified zones. In the latter case, 
students could choose only among the public schools located within their assigned zone. 
Whether delineated by zone or district, students could attend any school within that area, 
regardless of a school’s prior performance. This type of choice arrangement gave sites 
little or no ability to direct the flow of students from low- to higher-performing schools. 

At least one site deliberately defined its geographic zones to include both schools 
identified for improvement and higher-performing schools. Thus, in a large countywide 
district, “pie-shaped” zones could have suburban-like, higher-performing schools at their 
perimeter and urban-ltke, low-performing schools at their center. While the site limited 
transfers to within-zone choices, the initiative nevertheless gave students the opportunity to 
transfer from schools identified for improvement to higher-performing schools (and from 
an urban to a more suburban environment). At the same time, students also could transfer 
in the reverse direction. 

For both of these first two types of arrangements, the majority of the student bodies in 
all three types of schools (sending-only, receiving-only, or both sending and receiving) 
were nonwhite students and were eligible for the Free and Reduced-Price Lunch Program. 
However, in comparing the two types of arrangements, those with the nondesignated (both 
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sending and receiving) schools had lower proportions of low-income and minority students 
than the sending-only schools in the first type of arrangement; in turn, the receiving-only 
schools tended to have the lowest proportion of low-income students and Title I schools. 

Third, one site established a within-school initiative, in which students chose from 
education programs within the same school. In this third type, all students remained at 
their original schools. At the single VPSC site that implemented this third type of 
arrangement, students at their high school chose between two different academic programs 
that had been put into place with VPSC funds. Middle school students had a similar 
choice. At the elementary level, all students had the same academic program, but the 
students chose between two forms of assessment: being graded on a project or taking a 
test. 



Fourth, two initiatives involved a mixture of the first three types. One VPSC 
initiative had a mix of choices, including education programs within the same school, 
designated receiving schools, and schools that could be both sending and receiving. The 
second site under this last arrangement encompassed 26 school districts in the same state, 
with each district defining its own choice options. VPSC activities provided technical 
assistance to these districts in implementing public school choice. 



3.3 Core Activities in the VPSC Choice Initiatives 

Despite the unique circumstances of each of the VPSC initiatives, all sites focused on 
two core activities throughout the implementation process: 1) engaging parents and 
community members, and 2) building capacity at schools to attract and accommodate 
choice enrollees. 

3.3.1 Engaging, Notifying, and Reaching Parents and Community Members 

Sites undertook large-scale efforts to engage, notify, and reach parents and 
community members. From the beginning, the VPSC sites invested in outreach to parents 
and communities to ensure that public school choice initiatives met local needs. Sites 
started the outreach at an early stage. Sites even notified parents and community members 
of the plans for the choice initiatives to provide an opportunity for everyone to express 
preferences for the educational content, student selection criteria, and design of parent 
information centers (see exhibit 3-5). 

As the sites transitioned from the planning to implementation stage, several sites 
created ways to keep parents and community members actively engaged. For example, 
sites implemented information centers that sponsored workshops on parenting skills, health 
and nutrition, and computer literacy, as well as public school choice. 
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Exhibit 3-5 



Parent Involvement Activities 
Reported by School Officials in School Survey 



Type of involvement 


Sites’ activities 


Establishing the initiative 


Market research, surveys of parents, community and parent advisors, 
community focus groups and forums 


Planning the initiative 


Parent advisory committees, principai and teacher input, parent representation 
on iottery and parent information center committees 


Implementing the 
initiative 


Parent participation in daiiy instruction, community support for curricuium and 
materiais, parent and student surveys, speciaiists or counseiors, workshops 
on parenting skilis, and technicai assistance to schoois on parentai 
invoivement 


Community outreach 
activities 


Face-to-face parent and community outreach, media campaigns, direct maiiing, 
and parent information centers 



Exhibit Reads: Sites engaged parents in estabiishing the choice initiatives through market research, 
surveys, advisory teams, and focus groups. 

Sources: Analysis of site visit data and Grant Performance Reports, by COSMOS Corporation, 2007. 



To notify parents of their choice options and the details of the enrollment process, 
VPSC sites used a variety of outreach activities. Community outreach activities included 
face-to-face meetings, enrollment fairs, open houses at the receiving schools, media 
announcements, and letters to parents (see exhibit 3-6). Some sites communicated with 
parents and community members in multiple languages. Over half of the VPSC Program’s 
sites had brochures and applications available in both English and Spanish, and several 
reported advertising in local Spanish-language media outlets. One site reported that 
running an advertisement on a local Spanish-language radio station was among its most 
successful outreach efforts. Other sites reported printing materials in Chinese, Hmong, 

Lao, Polish, Portuguese, Russian, Somali, and Vietnamese. Public agencies also helped 
the sites to field phone calls from non-English speakers. 

During 2006-07, a large portion of school administrators at the VPSC sites indicated 
their belief that most or all parents and families were aware of choice options available to 
them (see exhibit 3-7). At the average VPSC site, 70 percent of the administrators reported 

1 3 

that all or most of parents and families had a good understanding of their choice options. 



Conducting a formal survey of parents’ perspectives was beyond the scope of the evaluation. 
However, parents were informally interviewed during the site visits. On those occasions, parents voiced no 
complaints about the choice options. 
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Exhibit 3-6 



Schools’ Reported Efforts to Notify Parents of Their Choice Options, 2006-07* 



Response category 


Average percent of 
schools per VPSC 
site** 


Individual, face-to-face meetings with school officials 


63.7 


Group meetings with school officials 


59.4 


Enrollment fairs or similar events 


60.5 


Open houses at receiving schools 


57.5 


Letter mailed to parents and families 


68.1 


Letter sent home with students 


57.0 


Announcements in community newspapers and other media 


63.6 


Contacts made by district’s parent information center(s) 


43.0 



Exhibit Reads: On average, for any VPSC site, 63.7 percent of schoois reported having individuai, 
face-to-face meetings between parents and schooi officiais reiated to choice options. 

*The School Survey covered 12 of the 13 sites in the VPSC Program because the 13th site focused mainly on 
technical assistance. 



**The averages represent the number of schools, not the number of parents, involved in these activities (the 
number of parents could not be determined); schools could appear under more than one category. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 



Exhibit 3-7 

Parents’ and Families’ Understanding of Choice 
Reported by School Officials, 2006-07* 



School-reported proportion 
of parents and families 

having a good understanding of their choice options 


Average percent of 
schools per VPSC site 


All 


25.7 


Most, over 50 percent 


44.6 


Some, 20-50 percent 


21.6 


Lew, less than 20 percent 


8.0 



Exhibit Reads: On average, for any VPSC site, 25.7 percent of schoois reported that aii parents 
and famiiies have a good understanding of their choice options. 

*The School Survey covered 12 of the 13 sites in the VPSC Program because the 13th site mainly focused on 
technical assistance. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 
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In some cases, sites delayed notifying parents of their choice options until their states 
published the names of schools identified as low-performing. As a consequence, parents 
and students at these sites were notified of their eligibility only a few weeks before they 
had to make a decision on whether to apply to transfer to another school. However, states’ 
deadlines for issuing this information have varied over the VPSC years, so the delayed 
notification was episodic rather than chronic at any given site. 

Over the course of their grants, sites shifted their outreach strategies and resources. 
One site reduced its districtwide marketing efforts but increased the schools’ involvement 
in the choice notification process. The same site also phased down its parent resource 
centers, previously VPSC-supported. Instead, the site upgraded its Web site to include 
more comprehensive choice information directly accessible to parents. It also encouraged 
schools to provide information directly to parents. 

3.3.2 Capacity-Enhancing Activities 

Sites used VPSC funds to enhance the capacity of schools. The sites participating in 
the VPSC initiatives supported a variety of capacity-enhancing activities, including: 
starting new academic programs; purchasing supplies and equipment; and providing 
professional development to staff. Capacity-enhancing activities included efforts to both 
accommodate and attract transferring students at receiving schools (see exhibit 3-8). In 
addition to improvements to existing schools, sites increased capacity within the system by 
opening new schools. For the most part, the new schools were planned in advance of the 
VPSC initiative, although VPSC funds later helped to start new academic programs at 
these schools. 

Activities to enhance school-level programming included focusing on professional 
development programs and education programs at all levels. The programs at one site 
included a foreign language program at the elementary level; an arts and academics 
program at the middle school level; and law, geographic information systems, and arts or 
design programs at the high school level. Across the sites, curricular topics covered by the 
capacity-enhancing activities covered academic subjects, such as language arts, science, 
and mathematics as well as research-based comprehensive programs such as the 
International Baccalaureate (IB) program (see exhibit 3-9). 

Although some of the new and enhanced education programs were nationally 
recognized with a record of improving student performance, in general, VPSC sites did not 
rely on scientifically based evidence to guide their selection of education programs. To 
date, the VPSC sites lack systematic documentation of such research for many of their 
capacity-enhancing education programs. 

Despite this approach, educational programming options implemented at the VPSC 
sites have received outside recognition. For example, an innovative science curriculum at 
one VPSC site resulted in new collaboration within a school district. The district utilized 
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Exhibit 3-8 



Capacity-Enhancing Activities 



Type of activity* 


Examples of programs supported 


Starting new schools or reopening 
older schools (4) 


Distance learning; charier schools; commuter schools; schools 
with thematic curricula 


Starting new or enhancing existing 
programs at schools (9) 


Literacy curriculum; mathematics and science themes; foreign 
language programs; environmental science enhancements 
for all academic subjects; International Baccalaureate 


Implementing tutorial and other 
support programs (6) 


During and after-school tutoring at school or by community- 
based organizations; assistance to supplemental service 
providers; hiring of support specialists 


Providing professional development 
to school staffs (12) 


Professional development workshops and summer institutes, 
covering choice, the transfer process, or new or enhanced 
curricula 



Exhibit Reads: Four VPSC sites have started new schoois or reopened oider schoois, inciuding a 
distance iearning schooi; charter schoois; commuter schoois; and schoois with thematic curricuia. 

*The number in parentheses represents the number of sites conducting each type of activity. 

Sources: Analysis of site visit data and Grant Performance Reports, COSMOS Corporation, 2007. 



the Web-based science curriculum in its entire set of K-3 classrooms. The VPSC initiative 
provided training and technical assistance in installing equipment and setting up the 
classrooms, as well as ongoing training and assistance to teachers. At another site, a strong 
environmental studies program — developed to attract children from low-performing 
schools to higher-performing schools — received a number of honors. For example, the 
VPSC initiative received the Governor’s Environmental and Economic Leadership Award, 
the state’s highest environmental recognition. 

Other conditions at the schools were important in encouraging enrollment in the 
choice initiatives. Existing academic programs, as well as other conditions at receiving 
schools, were important in encouraging enrollment. Other than VPSC-funded activities, 
administrators indicated that the preexisting academic programs were a major factor in 
students’ decisions to transfer. Similarly, interviewed principals, teachers, and parents 
overwhelmingly responded that the main reason students had decided to transfer to specific 
schools was their preexisting reputations for high performance. One VPSC site’s own 
parent survey confirmed the administrators’ conclusion that parents tended to choose 
schools according to the perceived rigor of academic programs. 
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Exhibit 3-9 



Illustrative Curriculum Topics at the VPSC Sites 



VPSC 

Site 


Capacity-enhancing activity 


Illustrative curriculum topics 


A 


Individualized, self-paced instruction using Arkansas Virtual 
School {ARVS) curriculum, derived from K-12, Inc- 
developed ourrioulum, using teohnology and distance 
learning for virtual learning environment. 


Research-based curriculum: Language Arts/English, 
Math, Science, History, Arts, Musio, Physical 
Education/Health 

Innovative K-3 Elementary Science 


B 


New oharter sehools. 


International Baccalaureate (IB) 

Junior Great Books (Language Arts), Connected 

Mathematics', FOSS (Full Option Science System), and 
the A History of U.S. series 


C 


Cluster schools implementing themes such as science and 
mathematics, fine and performing arts, international 
scholars, literature and writing, and world languages. 


Enhancements to curricula 
Lab improvements 
Clubs 

After-sohool educational programs 


D 


Receiving schools augmenting curricula, giving greater 
emphasis to grade-appropriate and technology-rich 
environmental education programs. 


K-12 interdisciplinary Environmental Studies Currioulum 
(enhanced by hands-on, inquiry based learning 
opportunities that integrate technology) 


E 


Site’s focus is on technioal assistance to distriots 


F 


Within-school academies with 250-300 students per academy 
and the International Baccalaureate (including IB Primary 
Years Programme). 


Humanities and International Studies (HAIS) 

Math, Arts, Science, and Technology (MAST) 

Exoelsior Humanities Aeademy 

Woodland Inventors, Technioians, and Scientists (WITS) 


G 


Attractor programs at schools, designed to enoourage urban 
students to select suburban schools. 


Themes offered are fine arts, oomputer teohnology, math 
and communication technology, aquatics, sign 
language, culinary arts, and environmental science 


H 


Schools are selecting academic themes with proven track 
records. 


Themes include pre-/6, literacy through the arts, pre- 
medicine, Waterford ear\'^ reading (literacy through 
media use), and college prep 


I 


Schools are offering supplemental programs to students. 


GEMS (Girls In Engineering, Math, and Science), ACE 
girls and boys programs, the Nationai Youth Sports 
Program, and the Learning Works program 


J 


Schools are opening unique programs of study available to 
any student in the area. 


The Academy of Arts and Design HS program 
The Law, Public Safety, and Security (LPSS) HS program 
The GIS (Geographio Information Systems) HS program 
Arts and Academios MS program 
World Language elementary program 


K 


Schools are developing and implementing themes and 
magnet programs. 


Foous on Literacy programs 


L 


Expanding higher-performing eduoational options and 
opening small learning communities. 


Arts and Teohnology 

Soience and Technology 

Young Women’s Academy 

Young Men’s Academy 

Elementary language immersion program 


M 


Schools are implementing enhanced programming for all 
students. 


Three faith-based after-sohool programs for tutoring and 
educational experiences 



Exhibit Reads: Site A used the ARVS curricuium as its capacity-enhancing activity. 

Sources: Analysis of site visit data and Grant Performance Reports, COSMOS Corporation, 2007. 
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Students also may have been attracted by schools’ physical facilities or perceived 
safety. The VPSC legislation prohibits the use of program funds for construction, but sites 
could use funds from other sources to make physical improvements. At one VPSC site, a 
newly reopened receiving school had a combination of new academic activities, supported 
by VPSC funds, and a new building and classrooms put into place by the school district. 

At another site, the planned receiving schools were new charter schools with a similar mix 
of funding sources. 

Other participating districts were already in the process of reconfiguring existing 
schools, opening new schools, and renovating schools. These actions did not cover all of 
the receiving schools. Nevertheless, the VPSC sites frequently cited these district 
activities as reasons for not undertaking specific steps to expand seat capacity at the 
schools. Many participating schools were also under-enrolled and could accommodate all 
transfer applicants. Furthermore, seat capacity was generally determined at the school- 
level, providing flexibility to allow for reconfigurations within classrooms to accommodate 
more students. 

Some of the sites actively examined potential options to expand available seats, but 
few implemented any specific changes. Some of the options the sites explored included; 
hiring more teachers, opening satellite learning centers, expanding online classes, adding 
new wings to existing schools, and building new facilities. One site modified the method 
principals used to assess their schools’ available seat capacity, which resulted in more 
accurate counts, but not necessarily more seats. Another site expanded its VPSC initiative 
to include additional schools, in this manner increasing the total number of seats available 
each year. 

Overall, however, the sites’ capacity-enhancing activities were not necessarily 
accompanied by an expansion of seats or classrooms at many of the receiving schools. 
For instance, none of the sites reported hiring more teaching staff or taking other steps 
simply to expand the number of seats to accommodate higher enrollments at the existing 
schools. 



3.4 Initial Implications for Federal Education Policy 
and Local Education Practice 

The discussion thus far has treated the VPSC Program like many other federal 
discretionary programs — tracing VPSC-funded activities at each site as if the sites were 
supporting discrete, federally funded projects. However, from the perspective of the VPSC 
Program’s role in informing federal education policymaking and local education practice, 
the reality at each site requires closer attention. First, as with many other school districts 
in the country, nearly all of the 13 sites already had a broad variety of public school 
options prior to the VPSC Program. Second, the VPSC Program permitted sites to 
enhance existing options, and sites did not need to start entirely new initiatives. 
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3.4.1 A Variety of Other Choice Initiatives 

New choice initiatives, such as those funded by the VPSC Program, did not occur 
on a blank slate. Public school choice has been available in many sehool distriets for 
deeades. Some of the most eommon ehoiee options have been: magnet sehools, 
opportunities to transfer to different sehools in a distriet, state-initiated ehoiee options, 
options for students with at-risk eonditions, and, most reeently, eharter sehools. Likewise, 
distriets with Title I sehools have offered ehoiee options sinee 1994. Overall, most 
distriets by now have a wide seleetion of ehoiee options. 

The experienees at the 13 VPSC sites did not differ. All of the 13 sites had existing 
ehoiee options prior to the VPSC Program. Some of the sites had a wide array of sueh 
options, with some students eligible to partieipate in two or more options. Although sites 
tried to make their entire array of ehoiee options work well, they did not neeessarily traek 
students’ partieipation on an option-by-option basis. For federal poliey, the implieations 
are elear. The benefits of investing in a partieular publie sehool ehoiee initiative may not 
be easily isolated, given the existing array of ehoiees available to students. 

Choice contexts in different states also can vary markedly. Going one step further, 
any expanded federal support for publie sehool ehoiee might more likely be administered 
on a formula basis through state departments of edueation rather than as direet, 
diseretionary awards to loeal sehool distriets or other loeal entities. The VPSC Program’s 
experienee, with three awards going to state departments of edueation, potentially preview 
some of the features of sueh a seenario. In partieular, the VPSC experienee showed that 
marked differenees existed among states’ ongoing ehoiee priorities. They varied from 
using federal funds to: provide teehnieal assistanee; meet the settlement terms of 
signifieant eourt rulings; or support an innovative, off-eampus ehoiee option (see appendix 
C, exhibit C-1, for more detail). 

The variations among these three states did not just refieet differenees in ehoiee 
preferenees. The variations refieeted eonsiderably different eonditions among states. 

Thus, any expansion of federal funding through state alloeations may have to antieipate the 
diversity of eontexts among states. 

3.4.2 Implications of the VPSC Program’s Main Condition of Award: 

Sites Could Either Support New Initiatives or Enhance or 
Expand Existing Ones 

Rather than speeifying any partieular ehoiee model, the VPSC Program gave sites the 
flexibility to define their own proeedures. Furthermore, the program permitted sites either 
to start a new ehoiee initiative or to “enhanee or expand” an existing one. The flexibility 
was important beeause sites eould eraft a new initiative but also work within their existing 
tapestry of options. 
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Acknowledging this complication is necessary for interpreting the findings in the 
remainder of this report. For some of the sites, the VPSC funds were only part of the 
support for the identified initiative. As a result, the contribution of the VPSC Program 
may have been overestimated. At other sites, the VPSC funds not only supported the 
identified initiative but also partially supported other choice options at the same site. As a 
result, the contribution of VPSC funds may have been underestimated. Unfortunately, the 
current analysis was not able to distinguish the extent of these over- or underestimations. 

Sites could use VPSC funds to partially fund preexisting initiatives. Most of the 
VPSC-funded initiatives enhanced existing choice arrangements, complicating any 
attempts to distinguish “new” from “old.” Enhancements meant that these same 
arrangements already had other sources of funding support and technical expertise. For 
example, four of the 13 sites used their VPSC funds to support existing, districtwide choice 
options (see appendix C, exhibit C-2, for more detail). 

Sites used VPSC funds to support different portions of choice initiatives. Most sites 
had a wide array of choice initiatives as part of their choice environment. For instance, 
some sites directed VPSC funds to services specifically supporting choice enrollment 
under a VPSC choice initiative, but the same sites also used VPSC funds to support other 
services related to all of the choice initiatives at the site (see appendix C, exhibit C-3, for 
more detail). 

This overall pattern — from sites that used VPSC funds as only a partial source of 
support for ongoing districtwide options, to sites that used their funds to support more than 
a single choice initiative — meant that the VPSC Program supported parts of an initiative at 
some sites and more than a single initiative at other sites (see exhibit 3-10). Because 
neither option can be disentangled, the analysis in the remainder of this report assumes that 
the VPSC Program was associated with a single initiative at each site, ignoring the 
situations where additional options were partially funded or subject to overlap [see 
variations (2) and (3) in exhibit 3-10]. 

The VPSC initiatives overlapped with Title I choice options. The VPSC Program 
started in the same year that federal legislation expanded support for Title I choice options. 
The legislation allowed spending up to 20 percent of Title I funds to support transportation 
costs when students wanted to transfer out of a Title I school identified for improvement. 
Three of the 13 sites defined their VPSC initiatives to coincide or overlap closely with 
their Title I choice options (see appendix C, exhibit C-4, for more detail). However, most 
VPSC sites defined schools participating in their VPSC initiatives as a broader set of 
schools than simply those designated under Title I as “identified for improvement.” The 
VPSC funds also could be used to cover the students’ transportation costs or services 
regardless of a schools’ performance status — i.e., whether identified for improvement, low- 
performing, or neither. 
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Exhibit 3-10 



Variations in Sites’ Uses of VPSC Funds to Support Choice Options 



( 1 ) ( 2 ) 



(3) (4) 



:VPSC; 



VPSC : :ypsc; 



; VPSC ; '.OTHER'. 



.' VPSC 



VPSC supports 
only a part of a 
choice option. 



VPSC supports a single 
option and also parts of 
other options. 



VPSC supports an initiative 
that overlaps with other 
choice options (e.g., Title I). 



The VPSC initiative 
is separate from 
other choice 
options. 



Exhibit Reads: Under Variation (1), VPSC supports only a part of a choice option. 
Source: National evaluation team. 
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4. PROMOTING EDUCATIONAL EQUITY AND EXCELLENCE 



Over the five years from 2002 to 2007, the VPSC sites enrolled many students in their 
choice initiatives. However, sites made less progress on other program priorities which 
also were relevant for assessing educational equity and excellence. 



4.1 Student Participation at the VPSC Sites, 2002-07: 

Eiigibies, Appiicants, and Enroiiees 

The exact number of participants in a choice initiative varied according to the 
definition of “participation.’’'* The evaluation tracked the following three possible 
definitions to measure participation: 

1) Eligible students: all students who could potentially participate in a 
VPSC initiative; 

2) Applicants: the set of eligible students who applied either to attend 
another public school or to participate in an academic program within 
their original school; and 

3) Enroiiees: those students who successfully applied and enrolled in a 
school or program as a result of a VPSC initiative. 

The VPSC sites reported data in each category annually. Their reports for 2005-06 
illustrated the wide variations among the categories for each VPSC site (see exhibit 4-1).'"^ 

All three categories of students can be considered participants in the VPSC Program, 
depending on the desired logic. For instance, the logic favoring a count of eligible students 
as participants, even if not applying to transfer to another school, is that these students 
indeed exercised a choice by deciding to stay at their original school. Narrower 
definitions would limit participants to include only applicants, or even further, to include 
only enroiiees. To permit readers to use their preferred logic, the national evaluation 
tracked and reported all three groups of students. The data also permit an estimate of 
participation rates, usually defined as the proportion of enrolling to eligible students. 

In 2005-06 and across 12 of the 13 VPSC sites, 24,921 students enrolled in the 
choice initiatives, reflecting an overall participation rate of 2.8 percent of the students 
eligible to enroll. The amount and rate of participation also appeared to reflect the type of 



See appendix C, exhibit C-5, for data eovering all five years, 2002-07. 

In 2005-06, the number of eligible students in the VPSC Program was more than 35 times the number of 
enroiiees, making eligibility the high end estimate of partieipation in the program. In some eases (see rows A, F, 
and FI in exhibit 4-1), individual sites defined a large pool of students as eligible, even though the initiative eould 
only aeeommodate mueh smaller numbers of enroiiees. 
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Exhibit 4-1 



Student Participation in the VPSC’s Initiatives, 2005-06 



VPSC Program 
sites 


Eligible 


Number of students 
Applying 


Enrolling 


A 


290,142 


214 


214 


B 


10,452 


170 


170 


C 


4,970 


446 


446 


D 


4,993 


114 


34 


E 


670 


NA* 


196 


F 


320,331 


NA* 


13,068 


G 


46,838 


1,620 


1,494 


H 


134,878 


NA* 


3,039 


I 


27,266 


674 


674 


J 


45,727 


5,249 


3,783 


K 


1,682 


1,682 


1,682 


L 


8,235 


NA* 


121 


M 


Site focused mainly on technical 
assistance to districts across the state. 


Total number of 
students 
Number of sites 
reporting 


896,184 

12 


10,169 

8 


24,921 

12 



Exhibit Reads: VPSC site A reported a totai of 290,142 eiigibie students. 

*Site did not track all applicants. 

Sources: Analysis of site visit data and Grant Performance Reports, by COSMOS Corporation, 2007. 



choice arrangement that each site implemented. The five sites that designated speeifie 
sending and reeeiving sehools showed a lower proportion of enrolling-to-eligible students 
than other arrangement types. In 2005-06, the pereent of eligible students who enrolled in 
these initiatives was 0.3 pereent. In contrast, at the five sites with geography-based 
arrangements where the same sehools were both sending and reeeiving schools, 3.8 percent 
of eligible students enrolled in 2005-06. (The 100 pereent partieipation rate for the 
“within-school” option is an artifaet of the absence of inter-school transfers.) 

Sites did not track applicants as consistently as they tracked other participant groups. 
Of the 12 sites reporting eligible and enrolling students for 2005-06, only eight sites 
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tracked and reported the number of applieants for their ehoiee initiatives. One site was 
unable to traek applieants due to an error in eoding VPSC applieations in the distriet’s 
enrollment database. In addition, in the last two years, several sites reported that 100 
pereent of their applicants became enrolled, even though they had reported lower 
partieipation rates in earlier years. The exaet reason for the 100 pereent rates is not known, 
but one possibility is that sites may have ehanged their reporting proeedures in later years. 

Calculating the number of enrollees also depends in part on whether the totals 
include enrollees from prior years. The typieal VPSC site traeked and reported the 
number of students enrolling in their initiatives eaeh year by eounting only the number of 
students who started enrollment in that year. Such “first-time enrollees” serve as a 
eonservative estimate of the overall enrollment in a ehoiee initiative. Another way of 
eounting enrollment involves ineluding enrollees from prior years. Thus, the more 
aeeurate estimate of enrollment in the VPSC Program’s initiatives, for the four years 
ending in 2005-06, fell somewhere between the 24,921 reported in exhibit 4-1 and a 
cumulative total of 49,616 first-time students enrolled in the initiatives since their 
ineeption in 2002-03 (see appendix C, exhibit C-5). 

Unfortunately, totaling aeross all years overestimates the total enrollment, whieh may 
have deelined due to attrition. A few sites made rough estimates of the attrition or 
“dropout” rates over the period of the VPSC Program, suggesting rates that ranged 
between 30 and 50 pereent. Several other sites estimated their repeat enrollments and 
eonjeetured eontinuation rates from 50 to 95 pereent. 

The rough estimates mean that the enrollment is likely to be far lower than the 
eumulative total of 49,616 students. However, exaetly how mueh lower is diffieult to 
determine beeause, even if the attrition rates were known, it is extremely difficult to 
classify the students who leave or drop out. Some students may have decided to enroll at 
yet newer sehools through the ehoiee initiative, while others may have returned to their 
assigned sehool and may be eonsidered true “drop-outs.” However, many students may 
have left the distriet entirely, not neeessarily having anything to do with their ehoiee 
experiences. An aeeurate estimate for eumulative enrollment would need to distinguish 
among these alternatives while also traeking those students that eontinued to attend sehools 
of ehoiee through the VPSC Program. 

Overall, the number of enrolling students in the VPSC Program increased during 
the earlier years of the program but declined in the program’s fifth year. Of the 12 sites 
enrolling students, only ten provided eligibility and enrollment data for four eonseeutive 
years, 2003-04 to 2006-07 (see exhibit 4-2). These data permitted an estimation of trends 
through the fifth VPSC year, 2006-07. The yearly data captured the number of first-time 
(i.e., new) enrollees eaeh year, not the total number that eumulated over all years. For 
these “first-time enrollees,” the trends showed that the VPSC Program at first averaged 
696 enrollees per site in 2003-04, then reached a peak of 2,459 per site in 2005-06, and 
then deelined to 2,167 per site in 2006-07. 



29 




Exhibit 4-2 



Participation Rates, 2003-07* 



Enrollment school 
year 


Average numbe 
Eligible 


r per site 
Enrolling 


Percent of eligible 
students enrolled at the 
ten sites with four years 
of enrollment data 


2003-04 


47,165 


696 


1.5 


2004-05 


54,009 


1,592 


2.9 


2005-06 


59,781 


2,459 


4.1 


2006-07** 


72,393 


2,167 


3.0 



Exhibit Reads: On average, VPSC Program’s choice initiatives had 47,165 eiigibie participants per 
site in 2003-04. 

*The averages were based on ten sites for whieh partieipant data were available for four conseeutive years. 
**2006-07 data are preliminary. 

Sourees: Analysis of site visit data and Grant Performanee Reports, by COSMOS Corporation, 2007. 



Participation rates also showed the same pattern over time, increasing and then 
declining from 2003-04 to 2006-07. For the same ten sites reporting data from 2003-04 
to 2006-07, the participation rates by first-time enrollees also increased from 1 .5 percent to 
4.1 percent, from 2003-04 to 2005-06, and then dropped to 3.0 percent in 2006-07 (see 
exhibit 4-2). 

The decline in total enrollment as well as in participation rates may reflect the actual 
saturation of the VPSC initiatives because a good (but unmeasured) portion of the earlier 
years’ enrollees still remained enrolled in the later years. Their continuing enrollment may 
possibly have limited the seats available for first-time enrollees in the final year. For 
example, an initiative that retains the same receiving schools from year to year will likely 
encounter limited seat capacity after several years of annual first-time enrollees. 

In addition, the supply and demand conditions in choice arrangements, from year to 
year, may be extremely dynamic, with the two conditions adjusting to each other in 
immeasurable ways. For example, one site discouraged applications in its last two 
reporting years, knowing that the available seats were limited. Shifts in eligibility pools also 
occurred over time. For instance, due to annual shifts in the identity and number of low- 
performing schools, the pools of eligible students shifted accordingly at three sites that linked 
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choice options to school performance. Four other sites expanded ehoiee options to additional 
grade levels, and one site expanded its targeted geographie area and thus inereased the number 
of schools covered. A fuller assessment of the partieipation in ehoiee initiatives, over a 
multiyear period, may require stronger reeordkeeping of all three kinds of partieipation 
(eligibles, applieants,'^ and enrollees) than the VPSC sites appear to have implemented. 

Regarding the demographie and aeademie eharaeteristies of the enrollees, not all of 
the sites kept preeise data, nor did they eompare the enrollees’ eharaeteristies with those of 
non-enrollees — students who did not participate in the VPSC initiatives. However, four 
sites did submit individual-level student data that were used to analyze student 
aehievement trends (see ehapter 5). These data ineluded demographie and aeademie 
eharaeteristies for the year prior to any VPSC participation. The data also included similar 
information from seleeted groups of non-enrollees (the sites’ selection processes also are 
deseribed in ehapter 5). 

Compared to the non-enrollees, the VPSC enrollees tended to: be slightly more white 
than nonwhite, have slightly lower partieipation in the Free and Redueed-Priee Luneh 
Program, and have slightly better aehievement seores in mathematies and reading. 
However, the differenees were not statistieally signifieant. Chapter 5 eontains all of the 
details about these data and their sources (see exhibit 5-3). 



4.2 Progress on Program Priorities 

The VPSC legislation had four program priorities. Three priorities were: to provide 
the widest variety of ehoiees; to encourage transfers from low- to higher-performing 
schools; and to provide opportunities for students to transfer to schools outside of their 
home districts. The fourth priority direeted sites to use some of their VPSC funds to 
support transportation serviees and eosts. These four priorities are diseussed next. 

4.2.1 Widest Variety of Choices 

The VPSC Program made progress on the first priority of providing the widest 
variety of choice. Sites expanded the assortment of ehoiee options in partieipating sehools 
and offered a large and diverse number of aeademie programs to transferring students. 

The enormity of the task should not be underestimated. At one site, applieants eomplete multiple 
applieations, one for eaeh sehool of interest. The site then makes seleetions among applieations, not 
applieants, and the reeeiving sehools aet like eolleges or universities who may have aeeepted applieants that 
have also been simultaneously aeeepted at another sehool. The desired reeordkeeping would not only have to 
distinguish between the number of applieants and the number of applieations but would also have to 
determine whether and where the student aetually finally enrolled. 

A similar pattern was found when the enrollees’ eharaeteristies were eompared to distrietwide 
averages at eaeh of the four sites. The parallel finding was important beeause non-enrollees were ehosen to 
mateh the enrollees (see ehapter 5), whereas the distrietwide averages represented all of the students at the 
four sites. 
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Some sites developed more school choice options than others, with as many as 12 
different options available. The VPSC initiatives also made efforts, through media 
campaigns and related activities, to increase parents’ awareness of the variety of education 
options available to them. 

The sites augmented the quality of choice options by implementing enhanced 
education programming at receiving schools. Several VPSC initiatives broadened the 
choices available to students through the improvement and development of thematic 
programming at participating schools. The programs implemented as part of the sites’ 
VPSC initiatives were new to the district (or a particular area of the district) and often 
distinct from other existing thematic programming. These programs focused on unique 
academic themes and were often integrated schoolwide, across all grades and subject areas, 
especially at the elementary and middle schools. Other programs, often implemented at 
the high school level, included courses focused on a particular theme that were separate 
from the required curricula. 

Three sites focused on adding new choice options as part of their VPSC initiatives. 
One statewide initiative implemented a virtual school, offering a completely new type of 
choice option to students, thereby directly increasing the variety of choice options within 
the state. At another site, magnet schools were the only form of school choice prior to 
VPSC. The VPSC initiative, by developing new charter schools, established a second 
choice option for students in the district. Lastly, a third site implemented a new choice 
school type. Lighthouse Schools, through their VPSC initiative, as an addition to the 
district’s other choice options. 

Two sites extended choice options of urban area students to a wider array of suburban 
schools. Another site permitted rural area students to transfer to a large number of 
adjoining districts, thereby broadening the students’ education options. 

4.2.2 Transfers from Low- to Higher-Performing Schools 

Sending schools tended to have higher proportions of minority and low-income 
students than receiving schools. Over 90 percent of the sending schools, on average per 
site, were Title I schools that served low-income students. Also on average per site, 72 
percent of the students in sending schools were nonwhite, and 74 percent were eligible for 
the Free and Reduced-Price Lunch Program (see exhibit 4-3). 



** At sites where geographie areas were used to foeus VPSC aetivities, thematie programming was new 
to that area although offered elsewhere in the distriet, and as a result, more aeeessible to students in that area. 

Examples of themes and programs inelude: math and seienee; International Baeealaureate program; 
world languages; fine and performing arts; literature and writing; humanities and international studies; 
environmental studies; dual-language; Spanish; eommunieations teehnology; eulinary arts; medieal skills; 
hospitality and tourism; Advaneed Plaeement elasses; and media industry. 
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Exhibit 4-3 



Demographic Characteristics of Schools, 2006-07* 



School characteristic 


n 


Avera 

Sending 
schools only 


ge percent per VP 

Receiving 
schools only 


•SC site 

Schools that are 
both sending and 
receiving 


Proportion of students: 

Race/ethnicity: nonwhite 


499 


71.7 


56.9 


72.6 


Eligible for Free and Reduced- 
Price Lunch Program 


592 


74.2 


59.0 


57.8 


Title I schools 


565 


93.1 


47.3 


75.3 



Exhibit Reads: On average (per site), 71 .7 percent of students in sending schoois were nonwhite. 

*The School Survey covered 12 of the 13 sites in the VPSC Program because the 13th site focused mainly on 
technical assistance. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 



The VPSC legislation favored initiatives that promoted the transfer of students from 
low-performing schools to higher-performing schools. The legislation used ESEA Title I 
provisions to define “low-performing” schools as schools not making academic yearly 
progress (AYP) two years in a row. Some of the sites had no Title I schools but used the 
accountability provisions within their states to define “low-performing” schools. In 
contrast, the sites did not specify explicit criteria for defining “higher-performing” schools, 
only that such schools could not be “low-performing” schools. Therefore, designated 
sending schools at these sites were schools identified for improvement and designated 
receiving schools generally had made AYP for two previous years (see exhibit 4-4). 

Transfers from low- to higher-performing schools comprised only a portion of the 
students enrolled in the VPSC initiatives. Five of the VPSC sites created choice 
arrangements with predesignated sending and receiving schools. Of these, only three 
limited their enrollment to transfers from low- to higher-performing schools in 2005-06 
(see exhibit 4-5, row A). Similarly, although another five sites permitted transfers 
throughout a district or a zone, only two tracked the portion of the transfers from low- to 
higher-performing schools (see exhibit 4-5, row B). 

Thus, aside from the five sites that either limited their transfers or tracked the 
transfers from low- to higher-performing schools, none of the other eight sites could 
provide such information (see exhibit 4-5, row C). At the five sites, the confirmed 
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Exhibit 4-4 



Reported AYP Status of Schools, 
2004-05 through 2006-07* 



AYP status in the past three years 
(2004-05 through 2006-07) 


Sending 
schools only 


No. (am 

Receiving 
schools only 


1 percent) 

Schools that are 
both sending 
and receiving** 


Total 


Made AYP 


9 


(21.4) 


21 


(77.7) 


114 


(34.8) 


144 


(36.3) 


Failed to make AYP 1 year 
Failed to make AYP 2 consecutive 


22 


(52.4) 


4 


(14.8) 


90 


(27.4) 


116 


(29.2) 


years 


9 


(21.4) 


2 


(7.4) 


86 


(26.2) 


97 


(24.4) 


Failed to make AYP all 3 years 


2 


(4.8) 


0 


(0.0) 


38 


(11.6) 


40 


(10.1) 


TOTAL 


42 


(100.0) 


27 


(100.0) 


328 


(100.0) 


397 


(100.0) 



Exhibit Reads: Nine (21 .4 percent) sending schoois made AYP for 2004-05 through 2006-07. 

*The School Survey covered 12 of the 13 sites in the VPSC Program because the 13th site focused mainly on 
technical assistance. 



**One VPSC site with a large, districtwide choice arrangement and many low-performing schools accounted 
for more than half of the schools that did not meet their AYP goals. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 



transfers from low- to higher-performing sehools (from rows A and B) represented 1,295 
of 5,927, or 21.8 pereent, of their total transfers in 2005-06. 

The other eight sites either permitted a wider variety of transfers or had VPSC 
enrollments that involved no transfers. Among the sites not traeking transfers, one 
reeeived a waiver from the Department to omit sueh traeking beeause, to be eligible to 
apply for transfer, all the students had to be low-performing (scoring “below proficient” on 
the state assessment) as well as of low-income backgrounds (eligible for the Free and 
Reduced-Price Lunch Program). At the same time, the actual transfers at this site were not 
necessarily transferring from low- to higher-performing schools. Thus, across the entire 
VPSC Program, the actual portion of low- to higher-performing transfers could be larger or 
smaller, depending on the nature of the transfers at the seven sites that did not track or 
document the pattern of their transfers (row C).^° 



The total number can nevertheless be used to estimate a range within which the current estimate of 
21.8 percent falls: Assuming all of the undocumented transfers were from low- to higher-performing 
schools, the proportion would rise from 21.8 to 81.4 percent; conversely, assuming that none of the 
undocumented were from low- to higher-performing schools, the proportion would dip to 5.2 percent. The 
range would therefore be 5.2 to 81.4 percent. 
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Exhibit 4-5 



Student Transfers from Low- to Higher-Performing Schools, 2005-06 











Transfers from low- to higher- 
performing schools 




Sites’ choice implementation 


No. of 
sites* 


Total 

enrollment 


No. 


Percent 


A. 


Only targeted students transferring from 
low- to higher-performing schools 


3 


650 


650 


100 


B. 


Supported various enrollments, and 
tracked the students transferring from 
low- to higher-performing schools 


2 


5,277 


645 


11 


C. 


Supported various enrollments, hut did not 
track students transferring from low- to 
higher-performing schools 


7 


18,994 


Sites did not track transfers from 
iow- to higher-performing schoois 


TOTAL 


12 


24,921 


1,295 

(minimum) 


unknown 



Exhibit Reads: The choice impiementation at three sites targeted students transferring from iow- to 
higher-performing schoois. The sites had a totai enroiiment of 650 students, aii of whom were 
transfers from iow- to higher-performing schoois. 

*Of the 13 VPSC sites, 12 reported enrollment in 2005-06. The 13th site has mainly foeused on ehoice- 
related teehnieal assistance. 

Source: Survey of Schools, COSMOS Corporation, 2006-07. 



In some cases, sites delayed notifying parents of their choice options until their states 
published the names of schools identified as low-performing. As a consequence, parents 
and students at these sites were notified of their eligibility only a few weeks before they 
had to make a decision on whether to apply to transfer to another school. However, states’ 
deadlines for issuing this information have varied over the VPSC years, so the delayed 
notification was episodic rather than chronic at any given site. 

4.2.3 Interdistrict Partnerships 

Most of the VPSC sites limited their choice initiatives to within-district options, 
rather than developing interdistrict partnerships. Although the implementation of formal 
interdistrict choice options might have expanded the variety of choices available to 
students even further, only five of the 13 VPSC sites created interdistrict options. At the 
eight remaining sites, many had existing transfer understandings with neighboring districts. 
However, these transfer options were separate from the VPSC initiatives and were 
generally reviewed on a case-by-case basis. Even though the existing options did not 
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include support for students’ transportation costs, the eight sites nevertheless did not use 
VPSC funds to promote or support interdistriet transfers at these sites. 

Among the five sites that did offer interdistrict choice options, one site used VPSC 
funds to support the opening of a eommuter sehool. The sehool provided an intra-distriet 
ehoiee option to the distriet’s students. In addition, the school accepted transfer students 
from other nearby distriets, thereby providing an interdistriet ehoiee option to students 
applying from other distriets. 

At the other sites with interdistriet ehoiees, the options varied from interdistriet 
21 

magnet schools to other formal transfer agreements with neighboring distriets. 
Interdistriet initiatives at these sites ineluded: I) regional interdistriet magnet school 
programs, in which magnet sehools aeeepted students from 26 sehool distriets; and 2) an 
urban-suburban transfer plan, whieh allowed urban students to enroll in any suburban 
distriet sehool and vice versa. Another partnership ineluded one urban sehool distriet and 
nine surrounding suburban distriets. Students from the urban district applying to transfer 
to any of the suburban sehool distriets were given priority in the applieation proeess. This 
interdistriet initiative also allowed suburban students to transfer to the magnet sehools 
within the urban distriet. 

4.2.4 Support for Transportation Services 

The fourth VPSC priority required that sites use VPSC funds to support student 
transportation serviees or eosts. This requirement reflected the assumption that students 
would be more willing to travel farther from home and eonsider sehools from a larger area 
than if the students did not have to pay the eost of transportation themselves. 

Relative to enrollment, transportation costs did not increase proportionately as 
might have been expected. This is partly beeause the VPSC initiatives permitted many 
students who were already attending distant sehools to seleet sehools eloser to home. 
Although nine sites reported using funds for transportation, overall these eosts did not 
neeessarily inerease. Many students already were attending distant schools, and the VPSC 
Program now allowed these students to seleet sehools eloser to home. Under these 
eireumstanees, sites experieneed minimal or redueed transportation eosts. 

Moreover, some of the VPSC Program’s ehoiee options eneouraged students, who 
otherwise might have been eontemplating a transfer to a more distant sehool, to remain at a 
neighborhood sehool. For example, one VPSC site identified neighborhood schools in 
which enrollment had deelined in part beeause of their pereeived poor quality. The site 
designated these under-enrolled sehools as “reeeiving sehools” and improved their 
edueation programs to eneourage ehildren to attend. As another example, two of the 
VPSC sites had reeently emerged from eourt-ordered sehool busing, from whieh students 



In 2004-05, one site’s interdistriet option ended with the closing of the magnet school. 
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had been assigned to more distant sehools as part of the original desegregation order. The 
VPSC-funded initiative gave affeeted students the ehoiee of returning to their 
neighborhood sehools. 

The sites generally used VPSC funds to supplement the distriet’s transportation 
budget, with three sites also implementing serviees speeifieally related to the VPSC 
initiative. Overall, VPSC funds supported new teehnology and personnel to improve 
transportation serviees. These improved serviees ineluded: a Global Positional System, 
mapping software for routing and dispatching school buses, transportation route 
coordinators, and support staff. 

Two initiatives also supported additional transportation activities beyond transporting 
students to school. These activities included busing students to before- and after-school 
activities, transporting parents to visit choice schools, and driving parents and students to 
VPSC-funded resource centers (e.g.. Parent Information Centers). 

In cases in which sites did not require additional funds for transportation, the sites 
received waivers from the Department to exempt them from the original requirement. One 
of these sites established off-campus receiving schools, which made transportation 
unnecessary for daily attendance. Other sites found it unnecessary to allocate additional 
funding toward transportation as other district funds were sufficient. 

In summary, the VPSC sites’ experiences demonstrated that transportation costs need 
not increase and may even decrease with the implementation of public school choice, 
depending upon preexisting enrollment patterns and the design of the choice initiative 
itself 



4.3 Implications for Federai Education Poiicy and Locai Education Practice: 
Further Issues of Educational Equity and Exceiience 

The findings in this chapter have directed attention to overall student participation in 
the VPSC Program and the sites’ progress in addressing the VPSC Program’s major 
priorities. However, an examination of “educational equity and excellence” can go beyond 
these issues. 

One type of additional examination pertains to potential lessons learned about the 
design of the various choice arrangements at the 13 VPSC sites. This is discussed next. 

To meet different equity objectives, different types of choice arrangements may be 
needed. Concerns over equity typically embrace a multi-faceted set of objectives: 
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1) Students, especially disadvantaged students (e.g., those of low- 
income status, or having low academic achievement), need to have 
greater opportunities to obtain quality education; 

2) Students of all backgrounds should have choices that allow them 
to match their own interests and goals with the variety of settings 
present in school systems; and 

3) Low-performing schools need to improve. 

Any specific choice initiative or arrangement may not be able to pursue all of these 
objectives simultaneously. In fact, the VPSC Program’s experience raises the possibility 
that most arrangements may only be able to focus on one or two of these objectives. 

A two-by-two matrix can start to represent these three objectives (see exhibit 4-6). 
The matrix also provides a way of assigning ten of the 13 VPSC sites’ initiatives (all but 
three sites could be categorized as having implemented “within-school options” or 
“mixtures” of different options). 

The two dimensions of the matrix specify whether eligibility to participate in a choice 
arrangement should be broad or restricted, with regard to either the eligibility of the 
students or the eligibility of the schools. These two-by-two combinations result in the four 
cells labeled “(A),” “(B),” “(C),” and “(D).” 

Cell (A) represents the broadest combination. It includes not only districtwide 
initiatives but also zonal initiatives. Although the designation of zones can have a 
potentially exclusionary effect, the two sites with designated zones did not exclude 
students who might have wanted to enroll within the zones but who came from sending 
schools outside of the zones (see appendix C, exhibit C-6, for more details). As a result, 
these initiatives had a “broad” representation of eligible schools despite their zonal 
character. 

Cell (B) represents a more limited combination, with restricted sending schools but 
with all students at these schools eligible to participate. However, the eligible sending 
schools do not need to be low-performing (see appendix C, exhibit C-7, for more details). 

Cell (C) also represents a limited combination. However, it differs from cell (B) 
because it limits the type of students eligible to participate, even though the students may 
be located at a broad array of eligible sending schools. Finally, cell (D) depicts the most 
limited combination: only a certain set of students, from a certain set of sending schools, 
are eligible to participate. 
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Exhibit 4-6 



Eligibility to Participate in VPSC’s Choice Arrangements: 
An Equity Perspective* 





Eligible sending schools 




Broad 


Restricted 




ALL STUDENTS AT ALL SCHOOLS: 


ALL STUDENTS AT SELECTED SENDING 




- districtwide initiatives (2) 


SCHOOLS: 




- muiti-school zones with mostiy zonai 


- students from designated sending 




but aiso districtwide eiigibiiity (2) 


schoois, but not necessariiy iow- 


o 




performing schoois (2) 


^ pa 




- students oniy from iow-performing 






sending schoois (3) 




(A) 


(B) 


S 


SELECTED STUDENTS AT ALL SCHOOLS: 


SELECTED STUDENTS AT SELECTED 




- disadvantaged students coming from 


SCHOOLS: 


s ^ 

‘S 


any schooi in the district (1 ) 


- none (0) 


X 

a 








(C) 


(D) 



Exhibit Reads: Two districtwide initiatives defined eiigibiiity to participate in a choice arrangement 
broadiy by focusing on aii students at aii schoois. 

*The number in parentheses in each cell represents the number of VPSC arrangements falling into a 
particular category. Not shown are the three arrangements that were either “within-school” arrangements 
or a mixture of arrangements. 

Source: Analysis of site visit data, by COSMOS Corporation, 2007. 



Within all four cells, the distribution of the ten VPSC initiatives shows the limited 
extent to which the initiatives ended up being restrictive, even in their design. Only one 
initiative confined eligibility to disadvantaged students, defined as students eligible for the 
Free and Reduced-Price Lunch Program and with lower academic achievement scores [see 
cell (C)]. Similarly, only three initiatives confined eligibility to low-performing sending 
schools, but within these schools the sites imposed no restrictions on the eligibility of 
participating students [see cell (B)]. The remaining sites all had unconstrained eligibility 
requirements, although the bulk of the students enrolling in these initiatives may still have 
been disadvantaged students. 

The matrix and the VPSC Program’s experience suggest that, if future policymaking 
and practice are intent on promoting transfers from low- to higher-performing schools. 
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choice arrangements may need to be more restrieted. For example, the VPSC Program 
might not just have eneouraged transfers from low- to higher-performing schools but might 
have limited transfers to those situations. However, limiting ehoiee to the most relevant 
eondition — the simultaneous eombination of a) low-performing sending sehools, and b) 
disadvantaged students [see cell (D)] — may be infeasible from the standpoint of loeal 
sehool polities and eommunity eohesiveness. Note that this most restrietive type of ehoiee 
arrangement was not found in the VPSC Program. 

To promote excellence, more guidance for capacity-enhancing activities may be 
needed. No matter what the ehoiee arrangement, an implieit assumption is that ehoiee 
students should enroll at a higher-performing sehool, if not a sehool of exeellenee. To 
boost the quality of educational programs at receiving sehools, all of the VPSC sites took 
steps to enhanee the capaeity of their reeeiving sehools. At some of the receiving schools, 
the eapaeity-building took plaee in the name of emulating edueational programs already 
found to be sueeessful at other sehools in the same district. However, sueh new adoptions 
did not neeessarily assure that the receiving sehool would become a high-performing 
sehool. 

One issue raised here is why the reeeiving sehools should have been ehanging their 
aeademic programs in the first plaee. Experienee with voueher programs for publie school 
students to attend private sehools seems to suggest the opposite. Beyond potentially 
reinforeing the ineoming students’ orientation and aeademie support, the reeeiving private 
sehools eontinue to adhere to their existing aeademie programs (e.g., see U.S. Department 
of Edueation, 2007, and the evaluation of the D.C. Scholarship Program). In fact, many 
private sehools have had suffieient sueeess with their aeademie programs that the sehools 
are almost rigid in adhering to their programs, even in the faee of external eireumstanees 
mueh more dramatie than participating in a voucher program. 

Eor the reeeiving sehools in the VPSC initiatives, one would assume that the greater 
ehallenge was to maximize the number of seats available in already higher-performing or 
exeellent sehools at the VPSC sites, not to modify in any way the schools’ existing 
aeademie programs. Yet, in the VPSC Program, every site provided extensive support to 
ehange aeademie programs at the receiving sehools, and these sehools appear to have 
enthusiastieally embraeed sueh aetivity (see appendix C, exhibit C-8, for more details). 

Under these eireumstanees, the VPSC Program might have offered more guidance on 
the nature of oapaeity-enhaneing aetivities, ineluding the presumed need to seleet programs 
supported by seientifieally based research. Absent any guidanee, some VPSC sites added 
to the aeademie programs of receiving sehools that had already been identified as higher- 
performing. Other sites started entirely new receiving sehools, charter sehools, off-eampus 
programs, or new magnet-like programs. Yet other sites strengthened programs at under- 
enrolled sehools to appear more attraetive as reeeiving sehools. 
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5. STUDENT ACADEMIC ACHIEVEMENT TRENDS 
CONCURRENT WITH THE VPSC INITIATIVES 



To assess student achievement trends, the analysis in this report traced student 
performance from the period prior to VPSC enrollment through the following years 
afterwards. These trends were then compared to those of a similar group of students who 
had not enrolled in any VPSC choice initiatives. 

Three cautionary notes must accompany the findings in this chapter. First, any 

observed student performance trends by the enrolling students cannot be readily attributed 

to VPSC initiatives. For example, more motivated students might have chosen to enroll in 

the VPSC initiatives than those who did not, and even though the analysis also includes 

data from comparison groups of non-enrolling students, the VPSC Program was not 

22 

designed as an experiment that might have ruled out such a self-selection bias. Thus, the 
observed student performance trends should only be considered as “concurrenf ’ trends. 
These trends have occurred during the period of any given VPSC site’s choice initiative, 

23 

but they have an unknown relationship to the initiative. 

The second cautionary note derives from the measures of student performance used 
throughout the analyses — scores on state achievement tests. The caveat in using these 
scores is that changed scores from year to year may reflect either changed student 
performance or changed procedures in scoring the states’ criterion-referenced tests. At the 
same time, the use of state assessment scores occupies a central role in major federal 
educational policies and especially the school accountability provisions under VCL5. 

Thus, the use of these scores in the following analysis appears justified, though with the 
appropriate caution. 

The third caution is that because the usable data were only available from a few of the 
VPSC sites, the aggregate analysis may not represent the VPSC Program as a whole. 



Without random assignment, any number of important but unobserved variables could account for 
differences between two groups. For instance, Cullen, Jacob, and Levitt (2005) studied school choice in the 
Chicago Public Schools from 1993 to 1995. The study found that “the leading explanation” for significant 
differences in graduation rates (students electing to transfer out of their neighborhood schools had higher 
rates) was that the transferring students were “superior along unobservable dimensions” (p. 755). High on 
the list of such dimensions was student motivation and the education background of students’ parents. The 
study concluded that, aside from transfers to career academies (emphasizing vocational skills and integrating 
school with work), “...systematic choice within a public school district does not seem to benefit those who 
participate” (p. 755). 

To overcome self-selection and other biases, one frequently raised possibility has been to use data 
from sites’ lotteries as if they were “natural experiments” (sites often use lotteries to select participants when 
the number of students exceeds the available choice opportunities). However, the lottery losers may not 
actually serve as the needed counterfactuals for lottery winners (see fuller discussion in appendix A). 
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5.1 Collection of Individual-Level Student Data 



All of the 13 VPSC sites were charged, from the beginning of their awards, with 
collecting and reporting records containing individual-level student achievement data. In 
principle, every VPSC site should have submitted individual-level data, coded to protect 
the anonymity of the actual student, that followed a template provided by the national 
evaluation (see exhibit 5-1). The template asked sites to define a student’s enrollment and 
demographic status, along with achievement scores over a multiple-year period. The ideal 
period would have started with the year prior to a student’s enrollment in a VPSC site’s 
initiative and then continued with annual scores for as many years as possible thereafter. 



Exhibit 5-1 

Desired Student Record 



Variables: 

1. Coded ID, but associated 


2001-02 


2002-03 2003-04 

Consistent Across Years 


Etc. 


with the same student 


2. Current Grade Level 


V 


y 


y 


y 


3. Current School 


V 


y 


y 


y 


4. Current District 


V 


y 


y 


y 


5. Race/Ethnicity 


V 


y 


y 


y 


6. Free and Reduced-Price 


Lunch Status 


y 


y 


y 


y 


7. Gender 


y 


y 


y 


y 


8. Achievement Scores 


y 


y 


y 


y 


(Math and Reading) 


9. Testing Grade 


y 


y 


y 


y 


10. Name of Assessment 


y 


y 


y 


y 



Exhibit Reads: A coded student iD consistent across years. 
Source: National evaluation team. 
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The template also asked sites to provide similar data for a non-VPSC group of 
students. The sites were asked to identify the eomparison groups in either of two ways: 
a) a matehed group of non-VPSC enrollees seleeted to mimie the demographie 
eharacteristics and baseline academic performance of the VPSC enrollees, or b) the entire 
set of students remaining at the sending schools from which the VPSC enrollees came. 
Regardless of the method chosen by a site, all subsequent analyses took into account any 
differences in the demographic and baseline characteristics of the VPSC and non-VPSC 
students before comparing their student achievement scores. 

Even if sites did not submit the needed data, they had the discretion to analyze data 
on their own and to report the results in their annual or final reports to the Department. 
Only one of these site -based reports was sufficiently complete to be reviewed as part of the 
national evaluation. The site’s findings are discussed later in this chapter. 

The desired databases could contain one or more cohorts of first-time enrollees and 
multiple years of annual data for each cohort. Because choice enrollments occur every 
year, each site also could have submitted data about more than one cohort of students. For 
instance, a site that had started enrolling students in its VPSC initiative in 2003-04 should 
have provided a baseline score (2002-03) and three additional years of data for the cohort 
enrolling in 2003-04. The site also should have provided another baseline score (2003- 
04) and two additional years of data for the cohort enrolling in 2004-05. In addition, some 
sites submitted data covering more than a single year prior to the onset of a cohort’s 
enrollment in the VPSC initiative. 

In response to the template, and given the possibility of multiple cohorts, sites began 
submitting data in 2005. However, the absence of key information in these submissions 
precluded their use, and the national evaluation recommended that the sites be given 
external technical assistance (by another contractor, not the national evaluation team) to 
collect and report the desired data. This assistance occurred during the sites’ fourth and 
fifth award years. 

Usable data came from four of the 13 VPSC sites, covering six of 38 potential 
cohorts of VPSC enrollees and comparison groups of students. By mid-2007, when the 
analysis for the current report needed to start, data from only four sites (and only covering 
six cohorts) contained the essential information. The sample of sites covered was therefore 
smaller than desired. Not only were the four sites a fraction of the 13 total sites, but the six 



However, whether the non-VPSC students later enrolled in a VPSC initiative, or otherwise transferred 
sehools in subsequent years, is unknown. Ideally, had the template been followed, the variable “eurrent 
sehool” would have provided information on a student’s matrieulation status in ensuing years. 

For example, one site only provided two years of aehievement seores, whieh were insuffieient to 
ealeulate the needed trends, as deseribed later in the text; a seeond site submitted eross-seetional data, only 
providing one year’s seores for eaeh sueeessive eohort of “first-time enrollees;” a third site’s state ehanged 
its assessment tests in the middle of the trend period, also then limiting the eomparable data to only two 
points in time. 
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cohorts (of first-time enrollees) were only a small fraetion of the total number of 38 sueh 
eohorts enrolled by the sites eolleetively, by 2005-06. 

Nevertheless, even though only six eohorts of data were available for the present 
analysis, sites appear to have made progress in eolleeting data, and this effort should 
eontinue with the extension of the VPSC Program beyond its initial five-year period. In 
partieular, the 14 new awards made by the program in 2007, also for a five-year period, 
inelude seven sites that had reeeived the original awards in 2002 and that then 
subsequently reeeived external teehnieal assistanee. For this reason, the potential for 
analyzing student aehievement data has improved, and any eontinuing evaluation of the 
program ean antieipate the availability of a larger and more representative set of data. 

The six eohorts in the analysis eame from four sites whose eolleetive profile was 
similar to that of the rest of the VPSC sites, as follows. As with most of the VPSC sites 
(see exhibit 3-1 earlier), the four all were urban sites with populations in the middle range 
(100,000 to 1 million). Similarly, the four sites mimieked the distribution of all of the 
VPSC sites with regard to being spread aeross all three enrollment size eategories, and 
being nearly evenly split between sites with higher and lower pereentages of nonwhite 
students and students eligible for the Free and Redueed-Priee Luneh Program. Finally, the 
four sites also evenly represented the two main types of ehoiee arrangements (pre- 
designated sending and reeeiving sehools; and sehools that are both sending and 
reeeiving — see exhibit 3-4 earlier). 

Aeross the six eohorts (see exhibit 5-2), the data eoverage and the starting years for a 
eohort’s first-year enrollment in a VPSC initiative varied (see exhibit 5-2, column s 2 and 
3). This meant that only three eohorts had two or more annual data points after enrollment, 
but the other three eohorts only had one data point. Moreover, one of these latter three 
eohorts had four data points prior to VPSC enrollment. The desire to retain as mueh of the 
data submitted by the VPSC sites aeeounted for this uneven eoverage, but with all of the 
trends re-eentered around the same “ti” (the year of first enrollment), regardless of the 
ehronologieal year. 

All of the six eohorts had both enrollee and eomparison (non-enrollee) groups. All of 
the sites ehose to define their eomparison groups by using some type of matehing 
proeedure. The sites reported these matehes in the following manner. 

For sites whose ehoiee initiatives involved predesignated sending and reeeiving 
sehools, the eomparisons were defined as: a) students remaining at the sending sehools but 
enrolled there for three years or less, matehed for gender, ethnieity, and English Language 
Learner (ELL) status; or b) students remaining at the sending sehools, matehed on grade 



A fifth site submitted data covering four cohorts, but the submission came well after the analysis was 
underway and could not be included in the analysis. 
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Exhibit 5-2 



Individual-level Data Submitted by VPSC Sites 



(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 






Year of first 


Total 




Number of records 






enrollment 


enrollees 




used in analysis** 






(and number 


reported 












of years of 


by sites 










Years of 


data after 


(see 






Matched 




annual 


first 


appendix 




VPSC 


comparison 


Cohort* 


data 


enrollment) 


C-5) 


Subjects 


enrollees 


students! 


A-1 


2001-02 to 
2005-06 


2005-06(1) 


170 


Math 

Reading 


170 

170 


161 

161 


B-2 


2002-03 to 
2005-06 


2004-05 (2) 


4,270 


Math 

Reading 


1,534 

1,526 


2,301 

2,295 


B-3 


2003-04 to 
2005-06 


2005-06(1) 


1,494 


Math 

Reading 


950 

938 


914 

913 


C-4 


1999-00 to 
2005-06 


2002-03 (2)*** 


501 


Math 

Reading 


89 

90 


722 

696 


D-5 


2002-03 to 
2005-06 


2003-04 (3) 


3,844 


Math 

Reading 


3,261 

2,974 


2,241 

2,198 


D-6 


2003-04 to 
2005-06 


2005-06(1) 


3,783 


Math 

Reading 


2,922 

2,351 


1,000 

634 






TOTAL 


14,062 


Math 

Reading 


8,926 

8,049 


7,339 

6,897 



Exhibit Reads: The first cohort comes from VPSC site “A,” and student achievement test scores 
were avaiiabie from 2001-02 to 2005-06. 

*Sites enrolled new students in choice options each year, and each year’s enrollees were considered “first- 
time” enrollees, whose achievement scores were tracked prior to VPSC enrollment and annually thereafter. 
Thus, in theory, a site operating a VPSC initiative for three consecutive years could have had three cohorts 
of first-time enrollees, with each successive cohort having one less annual data point. 

**Many of the enrollees originally reported by the sites had insufficient test data and could not be included 
in the final analysis. For instance, the enrollees included young elementary students who had not taken 
more than one year of tests at most. Other enrollees from higher grades might not have had three test 
scores because they were not tested each year. 

***This site collected data every other year between 2002 and 2006, so only three years of data were 
collected. 

|The matched comparison students were those not enrolled in the VPSC initiatives but who had similar 
demographic and academic characteristics as the VPSC enrollees (see text for discussion of sites’ matching 
procedures). 

Sources: Student-level databases submitted to the national evaluation by the VPSC sites. 
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level, gender, ethnieity, and eligibility for Free and Reduced-Price Lunch (FRPL). For 
sites whose choice initiatives involved districtwide initiatives involving schools that could 
be both sending and receiving, the comparisons were defined as: c) students from the 
same district, matched for gender, ethnicity, and eligibility for the Free and Reduced-Price 
Lunch (FRPL) program; or d) subgroups of comparison students by grade level, each 
subgroup having aggregate characteristics matching as closely as possible the transfer 
groups in gender and ethnicity. 

As a final note, the analytic preference for multiyear trends (as opposed to using a 
“pre-post” design requiring only two data points) reflected the nature of public school 
choice initiatives. Unlike the use of a new curriculum or classroom technology that might 
take place within a semester and whose “effects” might be anticipated immediately 
following exposure, choice options do not reflect discrete, time-limited “interventions.” 
Choice involves a change in educational pathways or school careers, and any likely impact 
on student achievement may only occur over a period of time. However, given the sites’ 
difficulties in reporting achievement data, three data points were considered the minimum 
number needed to calculate trends. The analysis therefore excluded sites that only 
submitted two years of data. Similarly, the paucity of multiyear data (especially 
following the first year of enrollment) precluded any examination of more subtle issues, 
such as whether students might have suffered an initial disruption but then performed 
better, two to three years after changing schools. 



5.2 Methodology for Analyzing Student Achievement Trends 

The analysis^^ used a meta-analytic strategy to aggregate scores across sites and 
produce findings for the VPSC Program as a whole. The main purpose of the national 
evaluation has been to assess the VPSC Program as a whole. The challenge is to arrive at 
cross-site findings. 

For the purpose of considering how to aggregate student achievement data across 
sites, one complicating issue is that the program did not specify how sites should 
implement their public school choice initiatives. Instead, sites were encouraged to define 
choice initiatives that suited their own local needs and circumstances. As a result, the 13 
VPSC sites all implemented different public school choice initiatives. Furthermore, most 
of the sites are located in different states; these states use different achievement tests to 
assess student performance. 



The known volatility of achievement scores was an additional numeric reason for requiring three or 
more years of data. Statistically, trends based on two data points assume no measurement error, which is 
unrealistic. 

The full methodology is found in appendix A, section 3. 



46 




These eonditions preeluded any direct aggregation of individual-level student data 
across the VPSC sites, necessary to satisfy the national evaluation’s need to arrive at 
findings for the VPSC Program as a whole. Instead, the site-to-site variations were better 
suited to a meta-analytic strategy. This strategy called first for determining the nature of 
the student achievement trends within each VPSC site, and then using these separate 
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findings as part of a meta-analysis to arrive at findings across the VPSC sites. 

Although the data and other conditions differed from cohort to cohort (and site to 
site), the initial, within-cohort analysis followed the same procedure, ultimately estimating 
a “mean effect size” for the changes among the students in each cohort. In other words, 
the analysis first estimated the achievement trends for each individual student. Then the 
analysis combined these into a group trend representing the entire within-site cohort of 
VPSC enrollees. Finally, the trends were compared to similarly derived trends for the 
matched comparison students. 

The meta-analytic procedure then combined all of the cohort-specific, mean effect 
sizes to estimate a “grand mean” effect size (e.g.. Cooper and Hedges, 1994; and Lipsey 
and Wilson, 2001). Such a grand mean effect size represents the aggregate “difference 
between differences” over time for all of the cohorts and sites, thus creating the needed 
program-wide benchmark for the VPSC Program as a whole. 

The main achievement trends were calculated on the basis of “scale” scores and 
“standardized scale” scores. Because the key evaluation question was to determine the 
student achievement trends occurring concurrently with enrollment in a VPSC-funded 
choice initiative, the main measure of a student’s achievement was his or her score on the 
annual state assessments. 

The specific assessment tests vary from state to state and therefore from VPSC site to 
site (except for those few VPSC sites that were located in the same state). Moreover, the 
assessments are usually criterion-referenced tests (CRTs) rather than norm-referenced 
tests (NRTs). Unlike national tests such as the Student Achievement Tests (SATs) 
administered by the College Board or the National Assessment of Educational Progress 
(NAEP), neither the state assessment tests nor their scoring metrics are comparable from 
state to state. Each state usually reports its students’ performance in two ways: 1) by 
giving a “scale” score — the specific numeric score attained on the test by a student; and 2) 
by indicating whether the student’s score exceeded the state’s desired level of proficiency, 
usually reflected by four or five categorical groupings of scores, such as “highly 
proficient,” “proficient,” “below proficient,” and “basic.” 

Investigators have used meta-analyses to examine many topies in edueation, ineluding: gender 
differenees in verbal ability (Hyde and Linn, 1988); the aehievement benefits of summer sehool (Cooper et 
al., 2000); and eomprehensive sehool reform (Borman et al., 2002). 

Effeet sizes are a measure of the magnitude of differenees between two eonditions — for instanee, the 
eross-seetional differenees in performanee between two groups, or the ehange over time for one group, or 
both. 



47 




Because the benchmark scores for achieving the four or five different proficiency 
categories may change from year-to-year in the same state, the present analysis only relied 
on the “scale” scores. However, the numerics of the scale scores (e.g., some being very 
large numbers versus others being small numbers, depending upon the test scoring 
procedure) also led to the need for “standardizing” these scores — converting them to a 
similar scale across tests and sites (see equation 1). 

Equation 1 

^ijk ~ ^jk 

z-score — 

Sjk 

i = individual student achievement score 

j = year 

^ = grade 

X = mean of all students’ scale scores for same year and grade level 

s = standard deviation of all students’ scale scores for same year and grade level 

Any potential biases from “standardizing” scores are not easily known. Therefore, 
preliminary analyses were conducted to examine both the “standardized” and “scale” 
scores, and the analyses determined that the results were consistent. The “standardized” 

3 1 

scores were then used in the remainder of this report. 

The initial preparation of the data included taking into account the demographic 
characteristics of the students as well as their baseline performance. Baseline 
demographic and academic differences are well-known influences or at least important 
correlates of student achievement outcomes. Any analysis of such outcomes, especially 
efforts such as the present one to compare the performances of different student groups, 
must take these conditions into account in order to make fair comparisons. As previously 
discussed, the sites already had matched a group of non-enrolling students to each group of 
enrolling students. Nevertheless, to create as close statistical equivalence as possible, the 
demographic and pre-VPSC achievement data were used to adjust the standardized scores 
and later achievement trends in the following manner. 

The VPSC sites provided only limited demographic data about the students. The data 
on the first condition covered different racial and ethnic groups, but because the definition 
of these groups varied from site to site, the information was condensed into two categories 
“white” or “nonwhite.” The sites’ data also covered a second condition — ^whether the 
student was eligible to participate in the Free and Reduced-Price Lunch Program (FRPL). 
Educational analysts commonly use FRPL eligibility as a proxy for a student coming from 



The analysis avoided another potential eomplieation that would have arisen if a state ehanged its 
assessment test during the multiyear period under study. No sueh ehanges oeeurred with the six eohorts 
under study. 



48 




a low-income or poverty-level baekground. The proxy measure is known to have 
inaeeuraeies and to mask other important differenees in students’ family baekgrounds (e.g., 
the edueation level of the parents), but the measure has been used extensively in researeh 
on student aehievement. Equation (2) shows how these eonditions were taken into aeeount 
in adjusting the standardized seale seores: 

Equation 2 



Z(SS) = brace * race + bpRPi * FRPL + cz{SS) 



Z(SS) 

race 

FRPL 

e 



standardized seale seore 

raee and ethnieity (white v nonwhite) 

eeonomie status indieator (eligibility for Free and Redueed-Priee Luneh) 
residual 



Following the adjustment for the demographie eonditions, the aehievement trends for 
eaeh student were then estimated using a simple linear or growth model. These models 
were then used to ineorporate a final step, whieh was to aeeount for differenees in students’ 
baseline aehievement seores, as these also ean affeet later outeomes (see Appendix A). 
Thus, the final analyses were based on growth models of students’ seores that had been 
standardized and that had ineorporated both demographie and aeademie baseline 
eonditions. 



Profiles of these baseline demographie and aehievement eharaeteristies, eomparing 
the VPSC enrollees with the non-enrolling students, showed differenees between the 
groups. As previously diseussed in ehapter 4, the enrollees tended to have a higher 
pereentage of white students, a lower pereentage of students eligible for the Free and 
Redueed-Priee Funeh Program, and better aehievement seores, eompared to the non- 
enrollees. However, none of the differenees was statistieally signifieant, either within or 
aeross the six eohorts (see exhibit 5-3). (For the purpose of eomparing the baseline 
aehievement levels, the “pereent profieient and above” was used for deseriptive purposes 
to give an idea of the level of student performanee in the two groups, rather than the 
standardized seale seores that were later used in the aetual trend analysis.) 



5.3 Findings on Student Achievement Trends 

The final student aehievement trends eame from six eohorts that ineluded VPSC 
enrollees and matehed eomparison groups of students, aeross four VPSC sites. The eross- 
site analysis first estimated the student aehievement trends for these two groups separately 
and then eompared the trends between the two groups. 

These two analyses served two different purposes. The first, estimating trends 
separately, was needed to establish whether either the enrollee or matehed eomparison 
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groups was alone moving in a positive or negative direetion. The goal was to determine 
whether the enrollees might have been performing worse, regardless of any relative 
difference between it and the comparison students. The second, estimating trends relative 
to each other, then captured the comparison between the two groups. (The two analyses 
involved two different units of analysis, students in exhibit 5-4 and cohorts in exhibit 5-5, 
and the values should not be compared in these exhibits.) 



Exhibit 5-3 

Baseline* Characteristics of VPSC Enrollees and Matched Comparison Students 



Baseline characteristics 


Pen 

(across 

VPSC 

enrollees 


cent 

six cohorts) 
Matched 
comparison 
students** 


A. Grade span: 






Elementary 


45 


54 


Middle 


35 


31 


High 


20 


15 


B. Race: 






White 


56 


48 


Nonwhite 


44 


52 


C. Free and Reduced-Price Lunch 


41 


49 


D. Proficient and above 






Reading 


74 


64 


Mathematics 


72 


61 



Exhibit Reads: Of the VPSC enroiiees, 45 percent enroiied in the eiementary grades in the 
baseiine year. 

*The baseline year is the year prior to a student’s enrollment in a VPSC initiative. 

**Students who were not enrolled in the VPSC initiatives but who had similar demographie and aeademie 
eharaeteristies as the enrolling students. 

Sourees: Student-level databases submitted to the national evaluation by the VPSC sites. 
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When the VPSC and non-VPSC trends were examined separately (see exhibit 5-4), 
the VPSC enrollees’ trends were neutral in math and positive in reading but not 
statistically significant. More important, the enrollee group’s scores were not found to be 
declining in any way. In contrast, the non-enrollee group showed a declining trend in math 

'X') 

proficiency that was statistically significant at the 99 percent confidence level. The 
reasons for the decline are unclear. Too little is known about the type of instruction or 
educational opportunities offered to the non-enrollees, either at the sending schools or for 
those sites whose non-enrollees came from different schools throughout a district. Finally, 
the non-enrollees’ trends for reading were positive but not statistically significant. 

Exhibit 5-4 

Performance of VPSC and Matched Comparison Students, Analyzed Separately 



Category 


Mean change 
per year 
in standardized 
scores* 


Standard 

deviation 


Intercept in 
standardized 
scores 


Standard 

deviation 


VPSC enrollees: 










-Math proficiency 


0.00 


0.35 


0.04 


1.31 


-Reading proficiency 


0.01 


0.35 


0.04 


1.29 


Matched comparison 










stndents: 










-Math proficiency 


-0.02** 


0.33 


0.04 


1.29 


-Reading proficiency 


0.01 


0.32 


-0.04 


1.25 



Exhibit Reads: VPSC enroiiees’ math proficiency changed an average of 0.00 of a standard score 
in one year. 

*A11 data are unweighted. Some groups’ data eovered three years and others eovered four or five. However, 
this small differenee was not assumed to ereate undesirable artifaets as might oeeur if the groups varied, for 
instanee, between three and twenty years. 

**p<.01 

Sourees: Student-level databases submitted to the national evaluation by the VPSC sites. 



Confidenee levels represent the probability that observed differenees do not differ by ehanee alone but 
represent aetual differenees between two groups. Statistieally, the analyses were based on the use of random- 
effeets models, whieh maximize the appropriate use of all of the sites’ data submitted for eaeh of the six 
eohorts. Unlike fixed-effeets models, the random-effeets models assume that the available data eome from 
but a sample of a fuller universe of students and that the sample therefore is likely to eontain additional 
varianee due to sampling errors (Bryk and Raudenbush, 1992). 
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When the VPSC and non-VPSC trends were compared, students enrolling in the 
VPSC initiatives had better student achievement trends than those not enrolling. The 
enrollees surpassed the non-enrollees in the trends for both mathematies and reading, with 
the differenees in mathematies being statistieally signifieant at the 99.9 pereent eonfidence 
level and in reading at the 99 pereent eonfidence level (see exhibit 5-5). The comparison 
was based on the meta-analysis of effect sizes across all six of the cohorts, with the 
individual effect sizes for each cohort having already accounted for the demographic and 
baseline differences between the VPSC and non-VPSC groups. Because the units of 
measure were standardized scores, the trends cannot easily be translated into everyday 
educational units, but the effect sizes (.020 and .028 for mathematics and reading 
respectively) appear to be modest. 



Exhibit 5-5 

Aggregate Comparison of VPSC and Matched Comparison Groups, 

Across Cohorts (n=6) 



VPSC and matched comparison 


Weighted mean 
gain for VPSC 


95 percent 
confidence 


z 


cohorts compared: 


cohorts 


interval 


score 


-Mathematics 


0.020 


0.0084 to 0.0311 


3.41** 


-Reading 


0.028 


0.0061 to 0.0388 


2.70* 



Exhibit Reads: The VPSC cohorts outgained the matched comparisons in mathematics by a 
standardized score of 0.020. 

All data are weighted according to each site’s variability (more variability means less weight). 

*p<.01; **p<.001 

Sources: Student-level databases submitted to the national evaluation by the VPSC sites. 



The values for effect sizes can range from negative infinity (extremely negative difference) to positive 
infinity (extremely positive difference), with most values falling between -4 and 4. However, the values are 
difficult to interpret with respect to their importance. Cohen (1988) offers one interpretation: that values 
from 0 to .30 represent “smaU” effects, from .30 to .60 “medium” effects, and from .60 to 1.00 “large” 
effects. In education, a large effect may be considered a difference of a grade level in performance (e.g., 
fourth-graders performing at a fifth-grade level). Nevertheless, attempted interpretations of effect size values 
for any given analysis, as with standardized scores more generally, are heavily affected by the specific 
sample sizes, variances, and other parameters in the specific data set being analyzed (e.g.. Cooper, 1981; and 
Rubin, 1992), and the present analysis was therefore unable to provide a precise interpretation of the effect 
sizes or standardized scores reported in this chapter. 
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The completion of this meta- analytic procedure required the data to be subjected to a 
test of heterogeneity, to determine whether excessive cross-cohort differences existed. If 
so, such heterogeneity would raise questions about the confidence or appropriateness of 
trying to use and interpret a single average or mean, to represent the collection of cohorts 
(the present analysis relied on a “grand mean” effect size to summarize the trends across 
cohorts). The results of the heterogeneity test show that the cohort-specific effect sizes 
were in fact too heterogeneous (or different) in reading, and nearly so in mathematics, to 
be adequately characterized by the grand mean as a summary statistic. The results of the 
heterogeneity test therefore suggest caution in calling attention to the enrollee and non- 
enrollee contrast. 

The heterogeneity results also raised a relevant substantive issue; The high variability 
among the individual effect sizes might have signaled some important differences among 
the actual choice interventions. As noted earlier, the four sites and six cohorts were evenly 
divided among the two main choice arrangements (pre-designated sending, and receiving 
schools and schools that are both sending and receiving). The possible heterogeneity could 
have been a split between these two types of arrangements — one arrangement having 
consistently larger effect sizes than the other. However, inspection of the data did not 
show such a split, neither type of arrangement showing an advantage over the other. 
Similarly, the data were inspected for an outlier effect — that one site or cohort might have 
been disproportionately influencing the grand mean. However, the data showed no such 
outlier effect or any other anomaly of note. 

The caution stemming from the results of the heterogeneity test needs to be 
accompanied by several other cautions about interpreting the differences in achievement 
trends between the enrollee and non-enrollee groups. The other reasons for exercising 
caution are as follows. 

First, and most important, in all of the choice initiatives, students can choose to enroll 
or not. As a result, an enrollee group may represent more highly motivated students or 
differ in other unobserved ways from a non-enrollee group, accounting for some or all of 
any subsequent differences in student achievement. In other words, without an 
experimental design, the designated non-enrollee group may not adequately represent the 
desired counterfactual condition for drawing a conclusion about the benefits of choice 
initiatives. 

A possible clue regarding the differences between the enrollees and non-enrollees in 
the present analysis is provided by the demographic and academic contrasts between the 
two groups, presented earlier in exhibit 5-3. The contrasts showed that the enrollees 



The conventional test of heterogeneity is a g-test. A significant Q statistic indicates that the 
individual effect sizes from each cohort are sufficiently different to be poorly characterized by a single 
summary statistic, such as the grand mean. In the present analysis, the results were nearly significant (hence 
possibly too heterogeneous) for mathematics {Q = 10.2, 5 df, p<.10) and significant (hence undesirably 
heterogeneous) for reading [Q = 20.56, 5 df, p<.001). 
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already were performing better than the non-enrollees and had a lower proportion of low- 
income and minority students, though none of these differences had been statistically 
significant. Nevertheless, without restrictions on those eligible to enroll, a choice initiative 
may permit higher qualified students to be among the enrollees. Such a possibility 
deserves careful examination in future research. 

Second, the trends in the present analysis, while representing a minimum of three data 
points, leaned in the direction of having more of the data points precede rather than follow 
enrollment in the VPSC initiative (see exhibit 5-2, column 3). Confidence about the trends 
concurrent with a VPSC initiative, much less inferences about any effects, would be 
increased if analyses were based on a larger number of data points following enrollment. 
Again, future research should try to garner a larger number of post-enrollment data points 
to reduce the interpretive problems. 

Third, and more generally, the data only came from a small sample of sites (four of 
13) and from only a small sample of the enrolling cohorts (six of 38 cohorts of enrollees 
across all of the VPSC sites by 2005-06). Data from more sites and more cohorts would 
produce a firmer set of findings about the program as a whole. The ongoing VPSC 
Program should put renewed emphasis on obtaining data from additional sites and cohorts, 
given the new round of five-year awards made by the program in 2007. The possibilities 
of such additional data are especially strong, given that seven of the newly awarded sites 
were continuations from the first round of awards. 

Finally, any continuing data collection and analysis effort should be accompanied by 
a modified procedure for defining the comparison groups, or non-enrollees. Sites should 
not try to match any particular group of non-enrolling students, as the sites did in the 
current evaluation. Rather, the more desirable procedure would be for sites to provide data 
from a larger but nonselected set of non-enrolling students, such as those students 
remaining in the sending schools or even a districtwide set of students. Analytic 
procedures such as propensity score matching or some similar procedure conducted by the 
evaluation team could then provide a fairer selection of matched comparison groups than 
the sites’ procedures in the current evaluation. 

Given all the cautions, the findings in this analysis of six cohorts from four VPSC 
sites offer early promise regarding the potential benefits of the VPSC Program. External to 
the present analysis, the local evaluator at a fifth site reported findings for two cohorts of 
enrollees in the fifth site’s choice initiative. The evaluation relied on cross-section 
comparisons only, not a trend analysis. The results were mixed, with the older cohort 
significantly outperforming a matched sample of eligible but non-enrolling students one 
year after enrollment, but the later cohort significantly uuJerperforming its matched 
sample one year after enrollment. These mixed results need to be interpreted with the 
same cautions as those discussed above. 
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6. USEFUL CHOICE PRACTICES AND OVERALL LESSONS LEARNED 
ABOUT VOLUNTARY PUBLIC SCHOOL CHOICE 



6.1 Useful Choice Practices for Future Choice Initiatives 

Key VPSC Program sponsors hoped sites eould demonstrate useful ehoiee praetiees 
for distriets across the country, to help them implement the Title I choice requirements of 
the No Child Left Behind Act. The VPSC sites and the national evaluation have 
identified an array of such practices and procedures. 

VPSC sites initiated choice-related practices potentially useful for other districts. 
The emerging practices covered a range of topics, including how to: design choice 
initiatives; involve parents and families; design capacity-enhancing activities at schools; 
and develop technical assistance procedures. Districts across the country can certainly 
learn from the program’s experiences and adapt specific choice practices to implement in 
their own communities (see exhibit 6-1). 

As an example of designing choice initiatives, one VPSC site demonstrated how 
intradistrict zones can be structured to include a heterogeneous group of schools. Students 
therefore have wide choice but minimize transportation time because they can stay within a 
zone. 



Similarly, as an example of a useful strategy for increasing parent notification, one 
VPSC site developed a user-friendly online database. Parents and students logged into this 
database and electronically searched for information about their choice options. Because 
many districts across the country have a large variety of choice options, the amount of 
information available to parents and students may be overwhelming. By organizing this 
information online, the site made it easier for parents and students to quickly learn about 
their options. 

Another VPSC site distributed hardcopies of materials to notify parents of their 
options. Over the course of implementation, the site converted its materials from a 
program-specific format (e.g., separate brochures for each choice option) to a school- 
specific format (e.g., separate brochures for each school, describing all of the choice 
options available at the school). The latter format allowed parents to collect information 
more easily about the programs offered at these schools, rather than having to search 
through materials about all programs, some of which were not relevant to the parents’ 
interests. 



For instance, see the statement by Sen. Joseph Lieberman (D-Conn.), Congressional Record, 107th 
Congress, No Child Left Behind Act of 2001, Conference Report, U.S. Senate, Dec. 18, 2001, S13413- 
S13414. 
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Exhibit 6-1 



Potentially Useful Choice Practices from the VPSC Program 



Design of Choice Initiatives 

Maximizing Choice Options Within Smali Geographic Areas: Defining subdistrict “zones” to inciude 
the most heterogeneous group of schoois possibie, so students can have wide choice but stili stay within a 
zone and therefore minimize transportation costs. 

Empowering Schools to “Own” a Choice initiative: Deveioping a formal process for soliciting 
schools’ participation, to encourage them to take greater ownership over a choice initiative. 



Parent Involvement 

Gaining “Input” From Families: Using large-scale surveys, media campaigns, and public forums to 
raise public awareness and participation in the choice initiative. 

Operating Parent Information Centers: 1 ) Designing and staffing multiple parent information centers 
to serve a large community; and 2) establishing a centralized, one-stop family resource center that handles all 
school- and choice-related procedures. 

Designing User-Friendly Communication About Choice Options: Creating easily searchable 
electronic or hardcopy materials, presenting students and their families with different kinds of choice options. 

Parent and Community Outreach Event: Organizing annual community event, showcasing all 
schools in the district, to publicize school choice options; successfully attracting families to attend by offering 
free transportation, free food, student performances, and translation services. 

Offering Parents Transportation Details to Facilitate School Choice: Allowing parents to 
incorporate precise transportation information into their school selection process by providing parents current 
bus route and stop maps, locations of stops closest to home, and estimates of student’s waiting and bus ride 
time, for each school option. 

Working Outside the System: Creating a “community movement” and mobilizing families to be aware 
of the performance of their schools, leading to the promotion of new choices. 



Capacity Enhancement 

Defining an Array of Educational Programs to Promote Student Diversity. Attending to the 
offerings at different schools to assure that the programs attract diverse student populations. 

Developing Formal Community Partnerships: Forming agreements with over 30 local institutions, 
including universities, businesses, and nonprofit agencies, to provide support for new educational programs 
at schools in the choice initiative. 

Expanding Distant Learning at “Off-Campus” Locations: Developing a K-8 curriculum for off- 
campus, online learning that is aligned with state standards and state assessments. 

Supporting Teacher Training Through a Master’s of Choice Program: Collaborating with a 
university to offer a M.Ed. degree program with a school choice concentration. 
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To further facilitate parents’ decisions, one site provided parents with transportation 
details at the time parents were reviewing their school choices. The district’s 
transportation director attended the district’s magnet school fair and provided bus route 
maps to parents. She worked with the parents in searching the transportation database to 
locate the bus stops closest to their homes. A computerized district map then identified 
potential stops and estimated a student’s bus ride time for each school option. For out-of- 
district students, the district also provided an “alternate stop” at daycare providers if the 
student planned to ride the bus five days a week. The transportation director interacted 
with parents and responded to their questions. As a result of all this, parents were able to 
incorporate accurate transportation information into their school selection process. 

Sites also engaged in creative capacity-enhancement efforts. For example, one VPSC 
site developed a broad-based set of partnerships with local businesses, community groups, 
and institutions of higher education (IHEs). These partners helped the district to 
strengthen new education programs. Moreover, the partnership, developed as a result of 
the VPSC Program, successfully garnered additional funds to expand existing educational 
programs further. 

One of the three state education agencies that received VPSC awards focused its 
initiative on providing technical assistance to districts as they developed and 
implemented their choice plans. This VPSC site established formal partnerships between 
mentor districts and mentee districts. All districts in the state were eligible to apply. The 
initiative organized 13 districts, which already had been implementing choice options, to 
serve as mentors. Each of the mentor districts adopted a mentee district that either had no 
choice plan or was at an early stage of developing a plan. 

The mentor and mentee districts worked together to develop the mentees’ choice 
plans and to share desired practices. Mentor districts also assisted mentee districts to 
develop effective parent involvement strategies and school choice Web sites. In addition 
to providing technical assistance, the mentor districts also used VPSC funds to expand 
their own choice initiatives. 

The mentors and the mentees both reported having benefited from the program and 
plan to continue the collaborative relationship beyond the end of the VPSC initiative. One 
complicating factor has been the stability of the mentor-mentee partnerships. Of the 13 
mentor districts, one mentor withdrew from the initiative after participating for the two 
initial years. Another mentor district, despite efforts to recruit a suitable mentee, has been 
without a mentee district for three of the four years of the choice initiative. As of 2006-07, 
12 mentor districts and ten mentee districts were participating, with two mentor districts 
not having a mentee. 

In addition, the same VPSC site established partnerships with two universities. The 
universities used VPSC funds to support choice centers that provided technical assistance 
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to districts statewide. Therefore, the initiative’s aetivities targeted not only the 
partieipating mentor and mentee districts, but also all the sehool distriets in the state. 



Distriets requested teehnieal assistanee by ealling a ehoiee center or by eompleting an 
online request form at the center’s Web site. The centers’ staff made site visits to all 
mentor and mentee distriets to assess the distriets’ needs and provide assistanee. The 
ehoiee eenter maintained a variety of online resourees, ineluding an online parent survey 
available to any distriet to eopy. The parent survey, as well as many other resourees, were 
available in English, Spanish, and Creole. The Web site also provided information about 
ehoiee-related parent resourees, edueator resourees, best praetiees, online eourses in family 
involvement, marketing materials, and professional marketing eampaign tips. 

Besides these useful practices, the VPSC Program produeed important lessons about 
publie sehool ehoiee. The final two seetions of this report discuss these lessons. 



6.2 Overall Lessons Learned from the National Evaluation 
of the VPSC Program 

The VPSC Program demonstrated a variety of public school choice options. 
Though limited to awards at only 13 sites, the VPSC Program showed how four types of 
publie school choice arrangements ean work in a diverse array of eommunities aeross the 
eountry. 

Among the legislative priorities, the arrangements added a wider variety of ehoiee 
options, eompared to what had previously existed at the sites. The VPSC sites’ 
experiences, as traeed throughout the earlier ehapters of this report, also have produeed 
important insights into the workings of voluntary publie sehool ehoiee. These insights 
inelude the following: 

• Choiee arrangements ean range from: a) having predesignated sending 
and receiving schools, to b) more flexible arrangements whereby the 
same sehools ean both send and reeeive, and finally to e) ehoiees 
among aeademie programs that take plaee entirely within the same 
sehool. 

• Some sites ean have any eombination of these arrangements 
simultaneously, ineluding options that meet Title I requirements. Sites 
ean therefore have a “tapestry” of ehoiee options, serving many 
students under many eonditions. 

• Measuring the extent of student participation in choice initiatives 
depends on whether the foeus is on eligible, applying, or enrolling 
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students; participation rates will vary depending upon the chosen 
measures. 

• Choice operations do not need to follow common stereotypes. 

- Large metropolitan school districts already may have diverse 
urban-like and suburban-like schools, mitigating the need for 
interdistrict arrangements or transfers; 

- Similarly, students may not need to travel greater distances to 
transfer to more attractive schools; instead, choice initiatives 
can try to make those schools closer to students’ homes more 
attractive. 

• The VPSC sites have developed a variety of procedures potentially 
useful for other local districts implementing choice options. The 
procedures include ways of gaining parent involvement and also ways 
of enhancing the capacity of receiving schools. One VPSC site also 
demonstrated an array of technical assistance practices for helping 
local districts. 

With regard to assessing student outcomes, sites’ efforts to collect the needed student 
achievement data improved over time, suggesting that the VPSC should continue 
examining student achievement trends as the program enters its second five-year phase. 
This last observation is especially pertinent because seven of the original awardees in 2002 
also were among those receiving new awards in 2007. Although this final report covers 
the first five years of the VPSC Program, its continuation beyond 2007 will provide the 
opportunity for additional monitoring and lessons learned about public school choice. 

The evaluation experience clarified desirable features to strengthen future 
program-level assessments of voluntary public school choice. The lessons learned from 
the VPSC’s cross-site and aggregate program-level experiences also help illustrate how 
future evaluations can potentially yield more conclusive findings about a choice program as 
a whole. Such evaluations would encompass the following critical features, also discussed 
throughout this report and in appendix A. 

• Careful recordkeeping and tracking of all participants. Sites should 
keep accurate and aggregate counts of all three types of choice 
participants (eligibles, applicants, and enrollees) and also should 



The lessons pertain to multisite evaluations, like the present one, in whieh the goal is to evaluate a 
program as a whole, not the workings at individual sites (the latter would still be the responsibility of 
separate project-\ewe\ evaluators). 
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compile individual student-level data for ehoiee enrollees’ 
demographie eharaeteristies and student aehievement seores. 

• More annual data points, covering a longer period after initial 
enrollment. Analyses of student aehievement trends should be based 
on a minimum number of data points eovering the years following 
enrollment in a VPSC initiative. For instanee, a minimum of three 
years’ aehievement data after enrollment would produee greater 
eonfidenee that trends had been eoneurrent with the VPSC initiative 
rather than with eonditions prior to the initiative. 

• Achievement data covering a larger proportion of sites and cohorts. 
Aehievement data from more sites and eohorts would produee a firmer 
set of findings about the program as a whole. The ongoing VPSC 
Program ean put renewed emphasis on obtaining sueh data, given the 
new round of five-year awards made in 2007. The possibilities of 
sueh additional data are espeeially strong, given that seven of the 
newly awarded sites were eontinuations from the first round of 
awards. 

• A modified procedure for defining comparison students. Sites should 
not try to mateh any partieular group of non-enrolling students, as was 
done in the eurrent evaluation. Rather, the more desirable proeedure 
would be for sites to provide data from a larger but non-seleeted set of 
non-enrolling students, sueh as those students remaining in the 
sending sehools or even a distrietwide set of students. Analytic 
procedures sueh as propensity seore matehing or some similar 
proeedure would then be used to seleet a matehed eomparison group 
from the non-enrollees. 

• Hypotheses about relationships to site profiles and activities. 
Evaluations need to speeify sueh hypotheses at the outset of a study. 
With suffieient data, sueh profiles and aetivities might inelude queries 
about the relationship between different types of ehoiee arrangements 
and the nature and extent of ehoiee outeomes. 

• Inclusion of all choice options at a site as part of the same evaluation. 
Most sites have a eolleetion or tapestry of options, serving slightly 
different goals. Together, all of the options represent a site’s publie 
sehool ehoiee initiative, and evaluating any single option alone does 
not represent the site’s underlying poliey goals (nor do sites 
neeessarily maintain reeords on an option-by-option basis). 

Evaluations should therefore embraee all of the options at a given site 
if possible. 
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6.3 Two Challenges for Improving Voluntary Public Schooi Choice 

Finally, the VPSC Program’s first five years also suggest a number of ehallenges for 
future improvement. Two of these challenges especially deserve attention by federal 
policymakers and local districts because the challenges deal directly with the central 
themes of educational equity and excellence. 

“Choice, ” by its nature, may need to cater to a broad array of potential student and 
family preferences. However, the broader the array, the more difficult for a choice 
initiative to limit participation to a narrow set of (low-performing) students or schools. 

The VPSC sites did not necessarily confront this potential dilemma. As previously 
highlighted in chapter 4 (see exhibit 4-7), some of the sites designed restricted choice 
arrangements (e.g., a small set of predesignated sending and receiving schools). However, 
the restrictions did not simultaneously cover “selected students at selected schools.” In 
contrast, other sites designed “broad” arrangements, either defining “all students at all 
schools” or “selected students at all schools” to be eligible to participate. 

Recalling that providing choice for all students is an equity objective, the lesson may 
be to place importance on acknowledging the need for a collection of choice options at any 
given site (see appendix C, exhibit C-9, for more details). Rather than encouraging sites to 
focus their attention on specific types of choice initiatives, the aim would be to assist sites 
in considering the most appropriate collection — some options appealing to all students or 
schools and other options limited to selected students at selected schools. Only in this way 
might the entire set of equity objectives be served. 

Similarly, future evaluations of public school choice might embrace all of the options 
at a given site, rather than focusing on a single initiative. Such a complete picture of the 
entire collection of options would be needed to provide a fair assessment of any equity 
outcomes. Moreover, the broader scope would complement those situations when sites do 
not themselves track participation in their options separately (i.e., option-by-option). 

The education programs within choice arrangements, not the arrangements 
themselves, help set expectations for the desired educational outcomes for students. In a 
certain way, choice arrangements, such as those supported by the VPSC Program, 
represent administrative arrangements. They aim to give students the opportunity to enroll 
in different schools or in different academic programs within a school. However, with the 
call for the VPSC sites to emphasize transfers from low- to higher-performing schools, 
sites were expected to go beyond promoting efficient and fair student choice and mobility. 
The VPSC Program also aimed to improve student performance. 
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Although the VPSC initiatives devoted sufficient attention to needed administrative 
procedures (e.g., notifying parents, soliciting and reviewing applications, and making 
school assignments), a somewhat surprising but consistent pattern among the VPSC sites 
was that they also strengthened education programs at the receiving schools. These 
education programs went well beyond the expected orientation or transitional programs for 
newly incoming transfer students. The programs included thematic programs (e.g., 
environmental studies or the arts), computer laboratories, curriculum enhancements, and 
other programs of study available to any student at the receiving schools. 

In principle, other than orientation and transitional programs for incoming students, 
the receiving schools in a VPSC choice arrangement should not have needed to install new 
or modified education programs. As with the private schools in the D.C. Scholarship 
Program discussed earlier, the preexisting programs at potential receiving schools already 
should have made them higher-performing schools, if not schools of excellence. 

Therefore, changes would not be expected, and although the original VPSC legislation 
permitted funds to be used for educational programming, the legislation offered no specific 
guidance about such programming. As one possible result, the newly implemented 
programs at the VPSC sites did not necessarily stem from any systematic inquiry into 
parents’ desires (see appendix C, exhibit C-10, for more details), nor did the VPSC sites, as 
noted earlier, follow any externally driven selection process, such as relying on 
scientifically based research to select the programs. 

More importantly, neither the VPSC Program nor any of the sites attempted to define 
a “higher-performing” school. Most of the sites defined such schools as those which were 
not “low-performing” schools. In other words, sites did not set any benchmarks for 
“higher” performance. Also absent in the VPSC Program has been any discussion of the 
standards for school excellence that should be maintained or monitored during the 
implementation of a choice initiative (see appendix C, exhibit C-1 1, for more details). 

These remarks are not intended, in and of themselves, as criticisms of the VPSC 
Program. The point is that the conditions make it difficult to set expectations for the 
student performance outcomes in a choice initiative. Students may be more highly 
motivated and work harder if they have been able to select their school of enrollment, but 
their educational outcomes still will be heavily influenced by the substance of the 
education at the receiving schools and their classrooms. Thus, students’ academic 
outcomes in choice arrangements may still be no better than the quality of the educational 
programs available to the students, regardless of the amount of choice that has occurred. A 
potential remedy is for choice arrangements to receive more guidance and clearer 
definitions and expectations for designating schools as “higher-performing” schools. 

These remedies would benefit future students in public school choice options. 
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1. INTRODUCTION 



The three evaluation questions were: 1) What are the eharaeteristies of the VPSC 
Program’s grantees; 2) How and to what extent does the VPSC Program promote 
edueational equity and exeellenee; and 3) What aeademie aehievement is assoeiated with 
the VPSC Program? To address these questions, the national evaluation followed a 
mixed quantitative and qualitative methods researeh design, with data eoming from a 
variety of original and arehival sourees. 

The evaluation was therefore a multisite, multiyear program evaluation, aimed at 
evaluating the VPSC Program as a whole. The program eovered 13 sites (nine distriets, 
three state edueation ageneies, and one nonprofit organization ) that reeeived VPSC 
awards from the U.S. Department of Edueation (the Department) in the fall of 2002. 

Timing of the initiatives. Some of the sites started enrolling choiee students as 
early as January 2003, but other sites earried out planning or other aetivities and started 
the following year or even later. Beeause all sites began some type of aetivity in 2002- 
03, the sehool year 2001-02 was eonsidered the base year prior to the implementation of 
the VPSC initiatives. The evaluation then traeed the sites’ aetivities for the duration of 
their five-year awards, through 2006-07. 

At the same time, onee having started the aetual enrollment of students, sites may 
have offered ehoice options to a new set of students every year. For eaeh eohort of 
students, the “intervention period” therefore varied. At the student level, this variation 
was taken into aeeount by aeeommodating different base years for eaeh eohort of 
students. A eonsequenee of this staggered pattern was that more years of annual data 
were available for the older eohorts than for the younger ones. 

Definition of the initiatives being studied. The national evaluation defined the 
initiatives under evaluation as the ones being supported by VPSC funds. The use of 
VPSC funds not only defined the initiatives of interest but also helped foeus on speeific 
faeets of the initiatives. For instanee, for some of the 13 sites, the bulk of the funds were 
used to support and enhanee parent information eenters or outreaeh and media 
eampaigns; yet others provided funds for new edueational programs at receiving schools 
to inerease their eapaeity and attraetiveness to serve transferring students. The use of 
funds also direeted the national evaluation’s attention to more speeifie aetivities, while 
still attending to the overall VPSC ehoiee initiatives at the 13 sites. 

Observations about the use of lotteries as part of an experimental design for 
studying voluntary public school choice. The use of random assignment lotteries, when 
ehoiee initiatives are oversubseribed, poses a ehallenging situation for edueation analysts. 
On the one hand, the existenee of the lotteries appears to provide a “natural experiment” 
for analysts interested in eomparing students partieipating or not partieipating in ehoiee 



^^The nonprofit organization operated charter schools and qualified as a school district, to be eligible 
for a VPSC award. 
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options. On the other hand, in order to conduet the analysis and fairly interpret the 
results, a host of obstacles must still be acknowledged and overcome. A tentative 
conclusion is that these barriers outweigh the apparent benefits of using the lotteries as 
“natural experiments.” 

First, the nonparticipants may have previously invested much effort before entering 
the lottery — such as visiting prospective schools and filling out applications for 
exercising choice transfers — all of which may have raised their expectations that then 
were dissipated by their lottery loss (not unlike applying to college, and then being 
rejected). Analysts have not studied the consequences of losing under these conditions, 
but analysts cannot assume that the losers’ subsequent status as nonparticipants 
represents the desired counterfactual or “no treatment” group. In the choice initiatives’ 
lotteries, those not selected to participate may in fact have been subject to an alternative 
and undesirable treatment: becoming a group of disheartened students having to remain 
at their original schools and thereby performing worse than they might otherwise have 
performed. 

Second, the lottery losers may not remain nonparticipants after their loss. The 
ongoing evaluation of the D.C. Opportunity Scholarship Program that provides public 
school students with vouchers to attend private schools (U.S. Department of Education, 
2007) has found that 56.6 percent of the lottery losers actually switched schools after 
having lost in the lottery (p. 6). Although only 4 percent actually switched to private 
schools, the rest did switch schools, presumably exercising a choice option of unknown 
sorts — but again undermining the students’ status as a “no treatmenf ’ group. Given such 
actual participation rates, the U.S. Department of Education report (2007) has therefore 
had to regard the lottery as an “intent-to-treat,” whereby a lottery win is recognized but as 
an offering of an opportunity (in this case of a sizable monetary scholarship) but not 
necessarily at all synonymous with actual choice participation (the treatment of interest). 
In fact, observing the lottery as an “intent-to-treat” calls further attention to the preferred 
counterfactual for making comparisons to students who exercise a choice option: the 
performance of students who did not even apply for the lottery but who stayed at their 
assigned schools. However, the stayers and movers are voluntarily formed groups, not 
subject to random assignment. 

Third, the VPSC sites were not organized to conduct or document their lotteries as if 
the sites were conducting research studies. The VPSC sites followed a number of 
procedures that minimized the ability to study their lotteries: a) they required multiple 
applications from the same student (one application per school, as in applying to a 
college or university) and conducted the lottery among applications, not applicants; b) 
they did not employ any follow-up procedure to determine where lottery winners and 
losers may have subsequently enrolled; and c) they neglected to retain lottery records 
because such records lose their administrative value immediately following the 
conclusion of a lottery. 



A-4 




2. DATA COLLECTION 



Overview. The relevant data ineluded; multiple site visits to the VPSC sites; 
surveys eovering an average of 50 sehools at eaeh VPSC site; and the eolleetion of 
arehival data about the partieipating students and the performanee of sehools in the 
distriet or the site as a whole. 

For the site visits, the instruments ineluded field protoeols to guide data eolleetion. 
The protoeols for the VPSC sites eovered implementation at the site, distriet, and sehool 
level. The sehool survey instrument was directed to principals of schools participating in 
the VPSC initiatives and to administrators of participating districts in one of the statewide 
initiatives. 

The instruments did not cover the collection and analysis of quantitative archival 
data to address the evaluation questions related to student achievement. Instead, 
individual-level student data were compiled and provided to the national evaluation by 
the VPSC sites. 

The completed national evaluation covered the entire five-year VPSC Program and 
drew from multiple sources of evidence: 

• Annual (2002-03 through 2005-06): 

— Original VPSC grant applications (summer 2002 only); 

— Grant performance reports from each VPSC site; 

— Individual records of participating students and 
a comparison group; and 

— School-level records of student achievement trends 
at VPSC sites and comparison sites. 

• Selected Years (2003-04, 2004-05, and 2006-07): 

— Field studies of VPSC sites and comparisons; and 

— Surveys of schools participating in the VPSC Program’s 
initiatives. 

Site visits. The site visits covered the VPSC project site as well as one sending 
school and one receiving school (or two participating schools at those sites not designated 
as specific sending and receiving schools). The sites chose two schools with a lot of 
choice activity (i.e., having many transferring students), so that the site visit team could 
observe and learn about the VPSC Program’s initiative in action. 

During all three rounds of field visits, a two-person team conducted the visits at 
each of the VPSC Program sites in the second semester of the school year (see exhibit 
A-1). The field protocols covered interviews with the VPSC’s project director and staff 
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at each site and with other key participants in the choice initiative, as well as the 
collection of documents and archival data in relation to the interviews. 



Exhibit A-1 

Schedule of VPSC Field Visits 





2003-04 


2004-05 


2006-07 


1. Arkansas Dept, of Education (Ark.) 


March 31 -April 2 


March 21-23 


May 8-1 0 


2. Brighter Choice Charter Schools (N.Y.) 


May 10-11 


April 7-8 


Jan. 17-19 


3. Chicago PubUc Schools (111.) 


Feb. 25-27 


Feb. 23-25 


Jan. 23-24 


4. Desert Sands Unified School District (Calif.) 


Jan. 26-27 


Jan. 31-Feb. 2 


Jan. 30-Feb. 1 


5. Florida Dept, of Education (Fla.) 


May 19-21 


May 2-4 


Feb. 27-March 1 


6. Greenburgh Central School Dist. No. 7 (N.Y.) 


March 9-1 1 


April 4-5 


March 20-22 


7. Hillsborough County School District (Fla.) 


Feb. 3-5 


Feb. 14-16 


May 1 -3 


8. Miami-Dade County Public Schools (Fla.) 


March 17-19 


April 27-29 


April 18-20 


9. Minnesota Dept, of Education (Minn.) 


April 21-23 


April 11-13 


April 24-26 


10. Monadnock Regional School District (N.H.) 


April 13-15 


April 4-6 


March 27-29 


11. New Haven Public Schools (Conn.) 


June 1-3 


May 10-11 


March 13-15 


12. Portland Public Schools (Oreg.) 


March 9-1 1 


May 10-12 


Feb. 6-8 


13. Rockford School District #205 (111.) 


May 3-5 


April 20-22 


Jan. 9-1 1 



Exhibit Reads: The Arkansas Dept, of Education was visited in the 2003-04, 2004-05, and 2006- 
07 schooi years. 

Source: National evaluation team. 



School survey. The school survey gathered data from schools participating in the 

38 

VPSC initiatives, mainly corroborating the participation in the VPSC initiatives by the 
specific schools. 

The school survey was based on a closed-ended questionnaire, directed at the 
schools’ principals. The questionnaire requested information about: 

• Student demographics; 

• School performance; 



Participating schools were those with students who were either eligible for or enrolled in a VPSC- 
lunded initiative. 



A-6 



• Choice options available to students; 

• Percentages of students taking part in these options; 

• Methods by whieh ehoiee information was being shared with parents; 

and 

• Whether staff members had been reeeiving professional development 

related to sehool ehoiee. 

At all but one of the VPSC sites, all of the sehools eligible to partieipate in the ehoiee 
initiative were ineluded in the survey. One statewide initiative was the exeeption: No 

schools were surveyed. Instead, the distriets with five or more students participating in the 

39 

ehoiee initiative were asked to eomplete the survey at the distriet level. 

The team used the following proeedure to obtain a high response rate. At the request of 
the sites, the evaluation team first gave the sites’ projeet direetors the questionnaires to 
distribute to the sehools, with a personalized eover letter from the projeet direetor requesting 
the schools’ participation. Schools were asked to fax the eompleted questionnaires direetly to 
the evaluation team, or to return the completed questionnaires to the site’s projeet direetor. 
The sites sent reminder e-mails to the nonresponding sehools approximately one week 
following the submission deadline. If neeessary, the sites then made follow-up ealls to the 
sehools. 

Subsequently, and following the efforts by the VPSC sites’ staff (and with the 
permission of the site), the evaluation team then made at least two rounds of follow-up ealls 
direetly to the nonresponding sehools, frequently re-faxing the questionnaire to the sehools. 
Using this proeedure, the team aehieved high overall response rates (see exhibit A-2). 

During the final round of administering the survey, the evaluation team distributed the survey 
to 689 sehools. Respondents at 630 sehools eompleted and returned the questionnaire, 
resulting in an overall response rate of 91 pereent. 

Performance reports. The data colleetion also eovered the VPSC grantees’ annual 
performanee reports submitted to the VPSC Program Offiee at the Department. Eaeh 
year the offiee issues reporting requirements, based in part on the data colleetion 
suggestions by the national evaluation team. After extensive review of grantees’ 
performance reports, the evaluation team suggested that the sites provide more detailed 
deseriptions of their VPSC-funded initiatives in subsequent annual reports. 

For instanee, the original reporting requirement had ealled for sites to report 
expenditures in ten categories: personnel; fringe benefits; travel; equipment; supplies; 
eontraetual; eonstruetion; other; indireet eosts; and training stipends. However, these 
eategories made it diffieult to determine how funds were used in relation to the ehoiee 
aetivities. For example, staff hired to develop new transportation routes and staff hired to 



At this site, the distriet was the appropriate respondent beeause many of the partieipating students 
were assigned to distriets but not to speeifie sehools, either prior to or as part of the VPSC initiative. 
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run a parent information center would both have been included under “personnel.” The 
evaluation team therefore suggested that the annual progress report include additional 
information, and that sites accompany their reports with a list of the sub-items used to 
compile the totals for each category (except for the “fringe” and “indirect cost” 
categories). The team further suggested naming the choice-related function (e.g., 
outreach; school capacity-building; professional development; student transportation; 
etc.) associated with each sub-item, rather than listing the name of the vendor or person 
who incurred the expense. 



Exhibit A-2 



VPSC School Survey: Number of Schools Surveyed and Response Rate 



Site 


2003-04* 


2004-05 


2006-07 


No. of 
schools 
in VPSC 


Surveys 

No. 


completed 

Rate 

(Percent) 


No. of 
schools 
in VPSC 


Surveys c 
No. 


jmpleted 

Rate 

(Percent) 


No. of 
schools 
in VPSC 


Surveys cc 
No. 


>mpleted 

Rate 

(Percent) 


A 


12 


8 


67 


12 


9 


75 


24 


14 


58 


B 






No participating schoois 






5 


5 


100 


C 


19 


19 


100 


46 


46 


100 


65 


59 


91 


D 


13 


13 


100 


13 


13 


100 


19 


19 


100 


E 


No participating schoois 


100 


96 


96 


Not participating in survey** 


F 


6 


6 


100 


6 


6 


100 


5 


5 


100 


G 


205 


113 


55 


202 


182 


90 


211 


202 


96 


H 


56 


56 


100 


58 


52 


90 


62 


57 


92 


I 


107 


92 


86 


147 


129 


88 


133 


127 


96 


J 


6 


6 


100 


6 


6 


100 


6 


6 


100 


K 


24 


22 


92 


19 


19 


100 


42 


27 


64 


L 


81 


58 


72 


94 


81 


86 


86 


79 


92 


M 


21 


17 


81 


21 


20 


95 


31 


30 


97 


TOTAL 


550 


410 


75 


724 


659 


91 


689 


630 


91 



*During 2003-04, the VPSC Program’s initiatives had not yet sealed-up to include 650 participating 
schools. 

**In 2006-07, the site did not participate in the survey, as the initiative focused on technical assistance 
(TA), not schools. Data related to TA were collected during the site visit. 

Source: National evaluation team. 
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3. METHODS FOR ANALYZING STUDENT ACHIEVEMENT DATA 



3.1 Overview 

The complete analysis of the VPSC data required three primary steps with multiple 
ancillary procedures within each step. Additionally, there were many decisions made 
about the data and the analyses along the way that may not be completely acknowledged 
within the body of the report. A brief overview of the three steps is now followed by a 
more detailed account. 

The first step involved the process of merging and recoding the data. Within that 
step, there were decisions pertaining to how certain variables would be recoded and 
treated when values were missing. 

The second step involved the primary within-site analyses whereby individual 
growth curves were computed for each student who provided sufficient data. Similar to 
the first step, there were many decisions. The decisions within the second step regarded 
the handling of missing data, centering scores at a specific time point, residualizing 
scores based upon available covariates, and estimating effect sizes. 

The third and final step involved the between-sites analysis. Data were aggregated 
to form composite scores by subject area (i.e., math or reading) and site. Once combined 
within subject and site, the effect size data were compared via a meta- analytic 
framework. Decisions about effect size weighing, outlier detection and handling, and 
data display were relevant at this stage. 

For the purposes of completeness and clarity, those steps and decisions are detailed 
below. The steps are outlined in a sequential format to guide the reader through the 
intricacies and decisions as they were made in the process. 



3.2 Data Management 

Merge raw datafiles into single, long format file. The VPSC sites had provided 
their data in a series of Excel spreadsheets. Each individual row in a table represented a 
single student from a single cohort and data for a single year. Thus, to get data from a 
site for a single cohort, data were extracted from the Excel tables and concatenated to 
form a multiple-record or long file format. After combining the data, each student had 
multiple rows in the data set. Each row represented a specific time point during the data 
collection process and contained the available achievement data. Once the long file was 
complete, the student demographics were merged by the student’s coded identification 
number to form a complete data set for each cohort and site. It is important to note that 
data were not combined between sites or cohorts at this time. 
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Recode numerical missing data codes. Once the data were arranged in a suitable 
format, eaeh value was eheeked to ensure that the range and eode were appropriate. 

There were some inter-site differenees with respeet to missing data eoding. For example, 
some sites eoded missing values with -9 exelusively while other sites used several 
different eodes — eaeh eode representing a different type of missing value. Regardless of 
the eode used, eaeh missing value eode was reeoded to a “not available” or “NA” 
speeifioation so that the statistics package would not treat the value as a legitimate 
observed value. 

Calculate key variables for subsequent analyses. Several variables that were 
instrumental in subsequent stages of analysis were eomputed at this point. One sueh 
variable was a dummy eode that represented whether the student ehanged sehools 
between years regardless of group elassifieation (i.e., enrolled or eomparison group). 

This eode was neeessary to ensure that a fidelity eheek eould be performed if neeessary. 

A seeond variable was eomputed and held in a separate data file that represented the 
number of repeated observations per student. This variable was instrumental in seleeting 
students who might be suitable for the growth eurve analysis. The new data file 
eontained the student identifieation number, eohort, and number of times the student had 
eomplete, valid aehievement data. Data from this table formed the basis for the next step 
of student seleetion. 

Identify students who possess sufficient data from both the enrolled and 
comparison groups. Students were retained for the analysis if data were provided for at 
least three (3) time points. The deeision to select only students with at least three time 
points was based upon the faet that the aim of the analysis was to provide both individual 
level trajeetories as well as group aggregated trajeetories. Fewer than three time points 
would have led to some (in some sites many) students with only suffieient data for 
estimating a gain seore. That is, two data points would have been suffieient for a ehange 
seore based upon two observed time points. The intereept and slope for that change seore 
would have been estimated without error sinee the line of best fit (i.e., the estimated 
growth eurve trajeetory) would be drawn through both points without any residual. 
Students who had only a single data point would not have provided sufficient data for 
anything but a single, point-estimate of performanee. Thus, for the purposes of 
eompleteness and eomparability, only students who provided a minimum of three data 
points were retained for subsequent analyses. 

Compute residualized scores from raw, standardized scores provided in the 
original data. Eaeh observed, raw standardized seore was residualized aeross all students 
to aeeount for raeial, ethnie, and soeioeeonomic faetors. Speeifically, the variables 
“race” (a factor coded aeeording to 0MB standards) and “eligibility for the Free and 
Redueed-Priee Luneh (FRPL) Program” were used to determine predieted values from a 
general linear model. Those predieted values were subtraeted from the observed values 
to form residual seores. The following equation represents the general formulation of 
those residualized scores {Cij). 



eij=yij-yij 
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where the subseripts i and j represent the student and time point, respectively. The 
predicted value iyij) was computed with the following general linear model: 



yij= YpraceraCCi + bfrpfrplij 

Note that the race variable consists of a summation operator because race was coded as a 
polytomous (i.e., multiple categories) discrete variable. The residualized scores by site 
will differ somewhat due to inter-site differences on the demographic characteristics. 

Transform the residualized scores into z-scores. Once the scores were residualized 
to account for racial and socioeconomic indicators, the residualized scores were further 
transformed into z-scores to allow for comparisons between time and site. The z-score 
transformation was essential because the data provided each year by each site contained 
different standardized achievement scores. For example, some sites provided data for 
some years that ranged from 50 to 200 whereas other years data from the same students 
may range from 200 to 800. Furthermore, grade was relevant to the transformation 
because the achievement tests often differ enough by grade to warrant a different 
transformed score. To eliminate those intra-site differences, the data for each year were 
transformed into z-scores by year and grade. The general approach to this z-score 
transformation used the following standard z-score equation: 

ziCijki) = 

Ski 

where the subscripts k and / represent the year and grade level of the student, 
respectively. It is important to note also that the transformation was done separately for 
each of the two educational domains of math and reading. Therefore, for completeness, 
the equation above may be further subscripted to indicate the achievement domain. For 
clarity and simplicity, that more complicated equation was not included. 

Clean data and ensure values are complete and available for the within-site 
analysis. The final stage of the data management process involved a thorough check for 
missing or miscoded values. Some sites had so few students at specific grade levels that 
the z-score transformation process resulted in either illogical or missing values. In those 
cases, there was nothing that could be done to retain those observations and therefore 
they were omitted from the remaining analyses. The process of identifying illogical or 
missing values was governed by sample size of grade levels, actual z-score magnitude, 
and missing data codes. In almost all cases where data needed to be discarded, the values 
were confirmed by hand to ensure that there were no computational or data handling 
errors. 
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3.3 Within-site Analysis 

The second step in the overall data analysis plan focused specifically on analyzing 
the final data set that was cleaned and coded in the previous step. After careful 
consideration, there was a decision made about the proper analytic technique for the 
within-site data. That decision was an individual growth curve model. It was reasoned 
that the process of reducing the data from the raw repeated observations into growth 
curve parameters would offer the best opportunity to capture the change process. 
Additionally, the options for computing growth curves include both OLS and ML 
approaches. Both have their advantages but only one seemed to fit best with the goals of 
reducing the repeated measures while maintaining as much variability in the parameters. 
In other words, there was an explicit goal to avoid any model-based truncation of the 
parameters. Those aims led to the selection of an OLS (i.e., individual regression) 
procedure instead of the more widely used ML methods found in software packages such 
as HLM and SAS. The primary difference between an HLM/Mixed model approach is 
that the level-one models allow for fitting of the growth curve and error covariance 
structures. Because the aims of the project were to use the growth curve parameters 
alone, the OLS models delivered what we needed and provided a clear, reduced data 
structure for secondary analysis. The complete process of within-site analysis is detailed 
below. 

Estimate individual growth curve parameter estimation. Each student that had 
sufficient data was analyzed using a simple linear model. The linear model was specified 
for each subject area resulting in two sets of growth curve parameters for each student. 
The linear model equation for the growth curves is as follows: 

ytj = biitimeij + boi + O7 

where is the observed value for student i at time j. The first parameter of interest is 
the bii because that represents the linear growth parameter over time {timeij ). Time was 
centered at the point when the VPSC Program began at each site. Thus, time was 
recoded from a calendar year to integers indicating the point at which the program had 
begun (i.e., “0” denoting the inauguration of the VPSC Program). Interpreting the slope 
parameter, a positive slope did not mean that the student necessarily had higher 
achievement. Because the scores were transformed at each year for each grade level, the 
positive slope indicated that the student increased her relative standing in the grade 
compared to previous years. All the values represented relative achievement 
performance — again this was necessary to counter the problems of non-commensurable 
exam metrics over time and between sites. 

Two other parameters are worth mentioning: the intercept (bob and the residual 
(eij). The intercept represents the student achievement point at which the student began 
the VPSC Program. Higher values indicate that the student began at high values relative 
to peers at baseline. The residual values show the efficiency of the prediction models by 
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student i and year j. Both the intereept and residuals are important for subsequent 
analyses and diagnosties. 

Estimate effect size. Onee the growth eurve parameters were available for eaeh 
student and for eaeh subjeet area, effeet size estimates eould be estimated. The effeet 
sizes were eomputed based upon the rate of ehange in z-score units from the intereept. 
Specifically, the equation for within-site, individual effect sizes was computed with the 
following formula: 



^within 



biitimei - 






where bn and boi are the individual growth curve parameters estimated in the previous 
step, timei is the time student i spent in the VPSC Program since its inception, and SE 
and are the standard errors of the linear and intercept growth curve parameters. 
These effect sizes are standardized values just like Cohen’s d but they are weighted based 
upon the number of values observed. More repeated measures meant more weight to the 
estimated effect size. Effect sizes were then estimated for random coefficient (often 
referred to as random effects) models. 



Aggregate effect sizes within site. Once random coefficient effect sizes were 
computed, the next step in the within-site analysis involved combining the effects to 
summarize the overall site effect of VPSC. Individual effect sizes were aggregated 
within site to produce a mean overall effect by subject area. The most typical way of 
aggregating these effects is by taking the measure of central tendency and estimating the 
variance about that measure. In short, the mean represented the average effect. Means, 
however, are adversely affected by outlier values and the effect sizes computed in the 
previous analysis produced a good number of values that fell well outside a believable 
range. Even the weighting process employed above did not protect against outlier 
estimates. Therefore, to treat the outliers, windsorized means were computed for each 
site and for each subject area. The windsorized means were based upon the innermost 99 
percent of the data. Within-site but between-group effect sizes were computed by using 
the standard method of computing Cohen’s d: 



d, 



dwE d^ 



wC 



between ' 




where d^E and d^c were the average, windsorized, within-site mean effect sizes ^ 

computed for the enrolled E and comparison C groups, respectively. Additionally, thc‘^w£' 
and values were the standard deviations computed from the windsorized, within-site 
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effect sizes for the enrolled and comparison. Once the between-group effect sizes were 
computed, further analyses were performed comparing the sites. 



3.4 Between-site Analysis 

The final major step in the VPSC analysis consisted of a between-site analysis. A 
meta-analytic approach was used for this step for a variety of reasons given in the main 
report. 

The final student achievement trends came from six cohorts that included VPSC 
enrollees and matched comparison groups of students, across four VPSC sites. The 
cross-site analysis first estimated the student achievement trends for these two groups 
separately and then compared the trends between the two groups. 

These two analyses served two different purposes. The first, estimating trends 
separately, was needed to establish whether either the enrollee or the matched 
comparison group was alone moving in a positive or negative direction. The goal was to 
determine whether the enrollee group might have been performing worse, regardless of 
any relative difference between it and the comparison group. The second, estimating 
trends relative to each other, captured the comparison between the two groups. Thus, the 
ideal positive result would be a finding that both groups had positive trends but that the 
enrollee group performed better than the matched comparison group. A less ideal 
finding, from a policy standpoint, would be that the enrollees performed better, but that 
both groups had negative trends. This less ideal situation would produce ambivalence in 
deciding, for instance, whether a particular educational intervention was desirable or not. 

The first, or separate group analysis, involved the following estimates: the average 
linear slopes across sites (labeled as the “mean change”), the standard deviations of 
those slopes, the average intercepts across sites, and the standard deviations of the 
intercepts. It is important to stress that these estimates all represented unweighted data 
across the sites. 

The second, or two-group comparison, involved the following estimates: the 
weighted mean gain of the VPSC enrolled sites relative to the comparison sites and the 
associated confidence intervals and statistics for those values. As a point of emphasis, 
the second analysis differed from the first because weighting was now applied (the 
greater the site’s variability, the lower the weight). 

The following steps were used to estimate the overall effects. All pooled effect 
sizes were weighted by an inverse variance procedure. For the random effects meta- 
analytic model, the DerSimonian-Laird estimate was used for weighting the effects prior 



The values represent the average ehange, per year, in standardized seale seores for eaeh of the 
groups. Some groups’ data eovered three years and others eovered four or five. However, this small 
differenee was not assumed to ereate undesirable artifaets as might oeeur if the groups varied, for instanee, 
between three and twenty years. 
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to pooling. The equation below provides an estimate of the heterogeneity varianee that 
constitutes the DerSimonian-Laird estimate of weighting pooled effects. 



x^-(n- 1) 



T = 



W,: - 






where Wi — Var(Oj) which is the inverse of the variance of the measure of interest in 
the /-th cohort and with /< /^. W; . These pooled 

effects, therefore, are weighted based upon the inverse of the variance of the sampled 
effects. The weights only become relevant if there is sufficient heterogeneity of effects 
between the sites. In the case of the present analysis, there was sufficient variability to 
warrant the weighting by this method. 

Conduct test of homogeneity. Summarizing disparate findings from sites only 
makes sense if those findings are reasonably homogeneous. That is, the sites must be 
viewed as sampled from a similar population where a measure of central tendency (e.g., a 
mean) suitably characterizes the effects for all sites. Testing for homogeneity in meta- 
analyses typically involves the g-test — a test developed by Cochran (1954) to assess the 
average deviation from the mean. The g-test can be calculated by taking the weighted 
sum of the squared effect size deviations from the mean effect size. Recall that effect 
sizes (d) were computed as the difference between groups for each subject area (see 
earlier effect size equation). The general formula for the Q statistic is: 

e = I - ff 

where T, represents the individual effect size estimates d computed in previous steps, T is 
the average effect size estimate computed for all studies, and Wi is the fixed-effect 
weighting factor for each study based upon the inverse variance of the study. The term Wi 
is defined as: 
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The statistic Q is chi-squared distributed, with degrees of freedom {df) defined as k 
-1 where k is the number of cohorts. Homogeneity is the null hypothesis, so values of Q 
that exceed the chi-square critical value are indicative of heterogeneity. It should be 
noted that the Q statistic is underpowered for small meta-analyses (i.e., very few effects 
aggregated) and overpowered for large meta-analyses. Regardless of these limitations, 
all meta-analyses rely on the statistic to assess whether homogeneity can be ruled out. 
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OMB Clearance #1875-0224 
Expiration Date: 10-31-09 

1. QUESTIONNAIRE FOR INDIVIDUAL SCHOOLS 
National Evaluation of the Voiuntary Pubiic Schooi Choice Program 

COVER SHEET 



Date: 
Respondent: 
Titie: 
Phone: 
Schooi Name: 
District: 
State: 

FOR NATIONAL EVALUATION USE 

ID Number: 

Date Reeeived: 



1 

School Survey, 2006-07 COSMOS Corporation 
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Fax: 301-215-6969 
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QUESTIONS 



1. School’s Name, Address, and Grade Levels: 
Name: 

Address: 



Grade Levels {circle lowest and highest): pre-K K1234567891011 12 

2. The following data were calculated based on student enrollment from which school semester? 

{check ONE only): 

for Spring 2006 

for Fall 2006 

for Soring 2007 

Other date (speeify the date) 



Total No. of Students: {no.) 

Ameriean Indian or Alaska Native: 

Asian: 

Black or African American: 
Hispanic: 

Native Hawaiian or other Pac. Isld.: 

White: 

Other; 



Eligible for Free and Reduced Price Lunch: 

With lEP; 

With Limited English Proficiency: 

Migrant: 



3. Does this School receive Title I assistance? Yes (schoolwide) Yes (targeted) N o 

If yes, has the school been identified as failing to make Adequate Yearly Progress, based on 
student achievement scores for the: 

for the 2006-2007 school year 

for the 2005-2006 school year 

for the 2004-2005 school year 

none of the above 
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4. School Choice Options for Students at this School: 

{check ALL that apply): 

a. students within the district may transfer to this school 

b. students outside the district may transfer to this school 

c. students may transfer from this school to other schools within the district 

d. students may transfer from this school to other schools outside the district 

e. Other - e.g., open enrollment {please 
explain): 

f. there are no school choice options {If you answered “f", please stop; do not 
respond to the remaining survey items.) 

{if you checked items a, b, c, or d): 

For 2006-07, about how many students have transferred ? 

g. (number of students transferring to this school) 

h. (number of students transferring from this school) 

i. Did teachers receive any extra staff or professional development, 

in relation to the transfer process? Yes N o 

]• If yes, what were the main topics of the staff or professional development? 

(topic 1) 

(topic 2) 

5. Notifying Parents about School Choice Options: 

What actions did the school take to notify parents/families of their choice options? 

{check ALL that apply) 

a. individual, face-to-face meetings with school officials 

b. group meetings with school officials 

c. enrollment fairs or similar events for parents to learn about choice options 

d. open houses at receiving schools 

e. letter mailed to parents/families 

f. letter sent home with students 

g. announcements in community newspapers or other media 

h. contacts made by the district’s parent information center(s) 

i. other {please explain): 

j. How many languages, other than English, have been used in these notification procedures? 

{no.) o 



School Survey, 2006-07 
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6. In your opinion, what proportion of the parents/families had a good understanding of their choice 
options last year? {check ONE only) 

a. all parents/families 

b. most parents/families (e.g., over 50 percent) 

c. some parents/families (e.g., between 20-50 percent) 

d. few parents/families (e.g., less than 20 percent) 



If you checked 6b, 6c, or 6d, what is the most important thing you can 
recommend, to 

improve parents/families’ understanding of their choice options? 




7. Has your school started new programs (e.g., magnets, academies, small learning communities, new 
academic subjects) to be more attractive, either to reduce the number of students transferring out or to 
increase the number transferring in? {check ALL that apply) 

a. becoming a charter school 

b. starting new magnets, academies, or small learning communities 

c. starting other new academic programs or subjects 

d. making other changes in school administration (e.g., changing school hours) 

e. other 

f no new programs 



Briefly describe the new programs and the main changes in school operation and 
administration. 




School Survey, 2006-07 
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2. SITE VISIT FACE SHEET 

(Data collection: 2006-07) 

Face Sheet To Be Completed for 
Each VPSC Site at the Time of the Site Visit 



Date of Site Visit: 
Site Visit Team: 



Name of Site: 

Lead Organization 
and 

Dept, for VPSC 
Initiative: 

Partnering 

Organizations: 

Type of Jurisdiction 
(e.g., rurai, suburban, 
urban): 

Student Popuiation: 

Ethnicity: 

White African Hispanic Asian Other 

American 



Site Visit Face Sheet, 2006-07 COSMOS Corporation 

Phone: 301-215-9100 
Fax: 301-215-6969 
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3. INTERVIEW WITH VPSC DIRECTOR AND SUPPORT STAFF 

(Data collection: 2006-07) 



1. Definition of the VPSC Initiative 

1.1 What activities or staff are being supported with VPSC funds? [Use the revised proposal budget as 
a starting point, but be prepared for changes to have occurred. ] 

1.2 Is it reasonable to define the initiative according to how the funds are being spent? If not, provide 
some other rationale for defining the initiative. [Also check the revised proposal budget.] 

1.3 Describe the VPSC initiative in 2-3 short sentences. 

1.4 How participatory was the planning process that was used in developing the VPSC initiative? 
[Vrobe mainly for parent and school participation, and describe the nature of the participation. ] 

1.5 What were the main problems encountered in planning (not implementing) the VPSC initiative? 

1 .6 What have been the main problems encountered in implementing the VPSC initiative? 

2. Identity of the Schools Involved in the VPSC Initiative 

2. 1 Define the breadth of the initiative across its participating schools: 



2.1.1 Academic years covered by implementation (not 
including planning year): 

2.1.2 No. and grade level of sending schools: 

2.1.3 No. and grade level of receiving schools or sites, 
by district: [Also see Qs. 7.1 and 8.1 below.] 

2.1.4 No. and grade level of remaining schools, by 
district: 

Within this number, no. and grade level of 

schools that might become sending schools in 
the near future: 

[Define criteria used.] 

Interview with the VPSC Project Director and Support Staff, 2006-07 COSMOS Corporation 
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2.2 Describe any “zone” pattern and how zones were defined. [If possible, obtain a map of the schools 
and of the zone pattern. ] 

2.3 How were the sending and receiving schools identified to be part of the initiative? [If related to 
school performance, obtain documentation of such performance. ] 

2.4 Were any logically eligible schools “exempted” from becoming sending or receiving schools? [If 
YES, please describe. ] 

2.5 What are the grades for which choice options can be exercised (e.g., the entering grade for each 
school, but not the other grades)? 

2.6 About how many students are eligible to transfer, due to the VPSC initiative, by grade level (and 
school, if possible)? 

2.7 About how many seats will be created at receiving schools, due to the VPSC initiative, by grade 
level (and school or site, if possible)? [Also see responses to Qs. 7.1 and 8.1 below.] 

2.8 How did the district estimate the number of seats available at receiving schools? [Also comment on 
the quality of the procedures and data used by the district. ] 

2.9 What is the schedule of tuition transfers, if any, that accompanied the student transfer? 



3. Student Assignment Criteria and Procedures 

3. 1 What criteria have been used to define students eligible for transfer, and what source of data is used 
to review whether students are eligible? 

3.2 How and when do the eligible students indicate whether they want to transfer, and how many 
potential receiving schools may they identify? [Obtain an application form and review a sample of 
the completed forms. ] 



3.2.1 Do they have to indieate, explieitly, that they 
wish to stay at their present sehool? 

3. 2. 2 How does the date when the VPSC choice 
applications are due coincide with the 
application dates for other choice options — 
e.g., to charters, magnets, or any other special 
schools? 

3.2.3 Identify the location(s) where the applications 
must be submitted and the rationale for 
selecting the location(s): 



Interview with the VPSC Project Director and Support Staff, 2006-07 
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3.3 What criteria are used in deciding to approve a transfer? {Confirm the criteria through 
documentation and discussions with others, including parents, if possible.] 



Check whether the following student assignment criteria Present Criterion 

are used; if YES, state the criterion: 

3.3.1 Proximity preferences (e.g., 50 percent of the seats are 

reserved for students who live near enough to walk to 
school: [Note whether these families still have to fill 
out a choice application.] 

3.3.2 Racial fairness guidelines (e.g., the ratio of Whites 

and non- Whites cannot vary more than 1 0 percentage 
points from that of the district as a whole): 

3.3.3 Socioeconomic fairness guidelines (e.g., the ratio of 

students eligible for free or reduced lunch cannot 
vary more than 1 0 percentage points from that of the 
district as a whole): 

3.3.4 /mg preferences: 

3.3.5 Special provisions, if any, for students requiring 

bilingual or special education programs: 

3.3.6 Substitution rules when available seats, especially 

those designated by proximity, race, or 
socioeconomics, are undersubscribed: 

3.3.7 Lottery or random assignment when seats are 

oversubscribed (and especially whether separate 
lotteries in effect exist for different subgroups of 
seats — e.g.. Whites and non- Whites): [If used, 
describe the lottery or random assignment 
procedure. ] 

3.3.8 Whether and how a transferring student will receive 

supplemental services: 

3.3.9 Whether and how an eligible but non-transferring 

student will receive supplemental services: 

3.3. 10 Whether the program maintains a waiting list, and 

how late in the semester students can still transfer, if 
a seat becomes vacant: 

3.3. 1 1 Whether the students originally enrolled in the 

receiving schools can be displaced: 
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3.4 Have there been any complaints regarding the student assignment process: 



3.4.1 By students and their families? \IfYES, 
describe.] 

3.4.2 By sending schools’ staff? [If YES, describe.] 

3.4.3 By receiving schools’ staff? [If YES, describe.] 

3.4.4 By any other relevant parties? [If YES, 
describe.] 



3.5 How has the district’s student population changed, relative to the population of school-age 
children? 



4. Parent Notification Procedures 

4. 1 How did parents participate (and how many participated) in the planning of the choice program and 
especially in the design of the parent information center? 

4.2 When were parents first notified of the choice options supported by the VPSC initiative? [Obtain a 
copy of the written notification.] 

4.3 What feedback information is collected to assure that all parents are being properly informed about 
their choice options? 



5. Parent Information Center(s) (PlCs) 

5. 1 What district-wide enrollment does the PIC cover, and how many students is it actually serving? 
[Depending upon the choice program, the number of applicants for transfer may only be a portion 
of the total student enrollment — obtain both numbers. ] 

5.2 What are the physical characteristics of the PIC’s location? 



5.2.1 Access conditions (e.g., parking and public 

transportation): 

5.2.2 Physical facility (e.g., in a high school; working 

atmosphere): 

5.2.3 Hours and days of operation: 

5.2.4 No. of different languages spoken by staff, 

and match to student population; 
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5.3 Enumerate the PIC’s staff, by title and FTE. Also note the supervisory relationships and to whom 
the PIC director reports in the district. 

5.4 Enumerate the outreach activities undertaken by the PIC during the past school year. Also, give 
some estimate of the frequency of each type of activity: 







How often? 


5.4.1 


Home visits or individual conferences with 
school (or choice) staff: 




5.4.2 


Use of mass media: 




5.4.3 


Use of community or neighborhood events or 
facilities: 




5.4.4 


Other: 





5.5 How have these outreach activities been adjusted, if at all, from year to year? 

5.6 How does the PIC know that its outreach activities are sufficient in reaching all eligible families? 
\Cite the actual data used by the PIC in developing this knowledge. ] 

5.7 What is the nature of the parent survey (sample and instrument), if any, conducted by the PIC, and 
how are the survey results used? 

5.8 How does the PIC work with individual schools? \IfYES to any of the following, please describe]: 



5.8.1 The PIC maintains up-to-date 

information about the schools’ 
enrollments: 

5.8.2 The PIC arranges tours of the schools: 

5.8.3 The PIC encourages schools to develop 

their own marketing materials: 

5.8.4 The PIC helps the schools to disseminate 

information about themselves: 
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6. Transportation Support within VPSC Initiative 

6. 1 Were VPSC funds used to support new transportation services? \IfYES, please describe.^ 



6.1.1 Purchase of buses: 

6. 1 .2 Hiring of drivers: 

6. 1 .3 Increases in drivers’ salaries: 

6. 1 .4 Revamping of technological 

infrastructure (e.g., computerized bus 
routes): 



6.2 Were school hours changed (e.g., staggered daily times or alternate school calendars) to control 
transportation costs? {If so, describe and note whether the changes were related to VPSC or not. ] 

6.3 Whether VPSC funds were used to support new transportation services or not, collect data prior to 
and during the VPSC Program on: [If possible, obtain these data for individual schools, especially 
the sending and receiving schools. ] 



6.3.1 The number of buses: 

6.3.2 The number of students riding buses: 

6.3.3 Average and range of trip times: 

6.3.4 The number of schools served each day 

by a single bus: 



6.4 How important are the augmented transportation services to the entire VPSC initiative (e.g., did 
parents pay for or otherwise support a portion of the existing services, before)? 



7. Capacity-Building {New Schools or Sites to Serve as Receiving Schools) 

7. 1 How many new schools or sites (and how many seats), if any, were opened as receiving schools, 
with VPSC support? {Also see Q. 5.7.] 

7.2 What are the substantive educational themes at these schools, and in what way do they reflect a 
diversification or replication of themes already covered by the existing schools? 

7.3 How were the themes chosen, and did parents participate in the process? 

Interview with the VPSC Project Director and Support Staff, 2006-07 COSMOS Corporation 
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8. Capacity-Building (New or Expanded Programs at Existing Schools or Sites) 

8. 1 How many existing schools or sites have been expanded or added as receiving schools (and how 
many seats), with VPSC support? {The total of the schools or sites and seats in 7.1 and 8.1 should 
coincide with the responses to Qs. 1.4.3 and 2.7 above.^ 

8.2 What are the substantive educational themes at these schools, and in what way do they reflect a 
diversification or replication of themes already covered? 

8.3 Describe the specific actions taken to increase the capacity of the existing receiving schools: 
[Check each of the following, and if YES, describe. ] 



8.3.1 New educational programs (e.g., magnets or 

academies within the school) at the 
existing schools: 

8.3.2 Administrative changes to increase the 

school’s enrollment capacity: 

8.3.3 Supplemental services to assist the 

transferring students: 

8.3.4 Professional development to assist educators 

in providing instruction to the transferring 
students: 

8.3.5 Other: 



9. Capacity-Building (Sending Schools) 

9. 1 Do sending schools lose funds in relation to out-transfers? [If so, describe the procedures and the 
amount of funds involved.^ 

9.2 Are the sending schools under a formal sanction category (e.g., in need of improvement, in 
corrective action, or restructuring), and how long have they been in the category? 

Interview with the VPSC Project Director and Support Staff, 2006-07 COSMOS Corporation 
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9.3 Whether in a formal sanction category or not, what actions will be (or have been) taken to 
improve the future performance of the sending schools (check all of the following)? 

9.3.1 Professional development or other 

educational assistance: 

9.3.2 Adoption of new educational practices: 

9.3.3 Restructuring of the school: 

9.3.4 Elimination of the school: 



10. Relationship to Title 1 Provisions of NCLB 

10. 1 Has the district started to offer the school choice options under the new provisions of Title I, and if 
so, how do these arrangements relate to those of the VPSC initiative? 



10.1.1 Are different schools involved in each, or 
are VPSC funds used to complement the 
use of Title I funds for the same schools? 



11. Transferring Students and Comparisons with non-Transf erring Students 

11.1 For each relevant academic year in implementing the VPSC initiative, obtain the following data 
(preferably noting the individual sending and receiving schools): 



11.1.1 How many students were eligible to apply 
for transfer? 

11.1.2 How many applied, and how many were 
transferred? 

11.1.3 What were the main experiences in dealing 
with under- or over-subscription? 

11.1.4 What were the main lessons learned 
about the transfer proeess, and did 
these lessons lead to modifioations in 
the proeedures used the following 
year? 
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1 1 .2 Based on the analysis of individual student records [such analysis may be done by the VPSC site], 
how do the transferring students compare to those eligible but not transferring, on the following 
characteristics? 



11.2.1 Annual academic performance: [Start with 
the year prior to the initiative.] 

1 1 .2.2 Demographic characteristics (White vs. non- 
White; school lunch vs. non-school lunch): 

1 1 .2.3 Other characteristics that might account for 
academic differences between the two 
groups (e.g., number of siblings; level of 
parents’ education; nature of parents’ 
employment and income): 



12. Sending and Receiving Schools and Their Changes Over Time 

12. 1 For each sending and receiving school and for each academic year of the initiative, track: 



12. 1 . 1 The total enrollments, by grade level: 

12.1.2 The proportion of White and non- White 
students: 

12. 1 .3 The proportion of students eligible for free 
or reduced lunch: 

12. 1 .4 The academic performance (mathematics and 
reading) of all the students at the school: 

[Use all test data available, even for grades 
not necessarily affected by the VPSC 
initiative, not just the transferring students.] 

12. 1 .5 Obtain district-wide averages for the above, 
omitting the sending and receiving schools, 
to permit contextual comparisons: 



12.2 At the receiving schools, what problems have emerged that might have been associated with the 
transfers or transfer process, and what remedies have been put into place? 

12.3 At the sending schools, what problems have emerged that might have been associated with the 
transfers or transfer process, and what remedies have been put into place? 
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12.4 What is the evidence that the sending schools have taken more serious measures to improve their 
performance or otherwise to become more competitive? 

12.5 Is there any evidence that potential sending schools are taking more serious measures to improve 
their performance or otherwise to become more competitive? 



13. Contextual Conditions 

13.1 What other choice options and programs are available in the district (not covered by the VPSC 
initiative), and how many students are involved in such options or programs? 

13.2 Have these other choice options and programs changed during the period of the VPSC initiative 
and might they have played a role in the preceding early outcomes? 

13.3 Have there been any notable changes by the surrounding districts that might have played a role in 
the preceding early outcomes? 

13.4 Have there been any changes in the community (e.g., residential relocation patterns) that might 
have played a role in the preceding early outcomes? 

13.5 How has the district’s “share” of the student population changed, relative to that of private schools? 
{Obtain annual data by grade level, if possible.^ 
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4. INTERVIEW WITH OTHER PARTICIPATING PERSONS 

(Data collection: 2006-07) 

1. What is your connection to the VPSC initiative [e.g., principal or teacher at participating school, 
parent of participating student, other]? 

2. How were you made aware of the VPSC initiative? 

3. Can you briefly describe your understanding of the VPSC initiative? 

4. What changes have you seen or experienced as a result of the VPSC initiative? 

5. How has student enrollment at your school [or your child’s school] changed during the past year 
and to what extent have these changes been associated with the VPSC initiative, compared to other 
conditions? 

6. What changes in the academic program have been made as part of the VPSC initiative, and how 
well are these changes working? 

7. What changes in school administration (e.g., redefining school hours or the school year) have been 
made as part of the VPSC initiative, and how well are these changes working? 

8. How smooth or disruptive was the timing or sequence of key events in the VPSC initiative — e.g., 
application deadlines or parent notification dates? 

9. How well do you think the VPSC initiative has been received by teachers, principals, and parents, 
and do you think they are sufficiently informed about any new role(s) expected of them under the 
initiative? 

10. How has student behavior and academic performance changed, and to what extent are these 
changes associated with VPSC? 

1 1 . Did VPSC provide any other benefits or create any other problems that have not been addressed by 
the preceding questions? 
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5. INTERVIEW WITH THE COMPARISON DISTRICT AND TITLE I STAFF 

(Data collection: 2006-07) 

Procedures 

One-day site visits will be made to comparison sites in spring of 2007 (February-May). Where 
the comparison sites are within the jurisdiction already covered by the VPSC Program’s site (e.g., 
another “zone” in a VPSC district or another district within a statewide VPSC initiative), the needed 
data should be collected from the VPSC and related district staff. 

Where the comparison sites are outside of the VPSC jurisdiction, initial contact with the sites 
should emphasize that our National Evaluation is: 1) covering public school choice, 2) selecting some 
sites where minimal choice options other than Title I, magnets, and charters are in operation, and 3) 
making site visits to sites that are near a VPSC Program’s site but that did not receive a VPSC award. 
To minimize burden, the site visit will involve interviews with: the Title I coordinator and any related 
district staff (e.g., overseeing magnets and charters). 

Preparation for Site Visits. Prior to any site visit, field teams should collect and analyze data 
about the targeted area. This information will have come from several sources, including: the VPSC 
Program’s site that is aware of (and helped to select) the comparison site, and materials gathered from 
the district or state Web sites. 

Assembling of Evidence and Preliminary Reports Immediately Following Site Visits. Field 
teams are urged to begin the formal analysis and report-writing process as soon as a site visit has 
ended, though additional data may still have to be collected. Assembling data and drafting narratives 
proceeds more efficiently and with much higher quality if this time sequence is followed. Teams 
should reserve the day or two after the site visit for this activity, avoiding other commitments. 

Outline of Report. The report should follow the same heading structure as the topics of inquiry 
described next. 



Topics of Inquiry 

The topics below not only cover the “agenda” to be followed by the field team but also give 
explicit probes and examples regarding the type of evidence that is being sought. As a result, the 
protocol should provide guidance on how to know what to look or listen for, and how to recognize 
relevant evidence when it is encountered. 
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1. Public School Choice Options at This Site 

Enumerate the public school choice options at this site, giving the number of schools and students 
involved by type of option: 

1 . 1 School identified for improvement under Title I: 

1.2 Magnet Schools (no. of schools and programs) : 

1.3 Charter Schools: 

1.4 Other choice arrangements, including unsafe school choice and desegregation {describe): 

[If options were identified under 1.2, 1.3, or 1.4 above, be sure to cover Q. 3 below.) 



2. New Title I Choice Arrangements 

How has the district started to offer the school choice options under the new provisions of Title I? {probe 

for): 



2. 1 Use of designation as a Title I school identified for improvement in defining sending and receiving 
schools: 

2.2 Expanded transportation services: 

2.3 Provision of supplemental services: 

2.4 Expanded or improved parent notification procedures: 

2.5 Other (describe): 

3. Other, non-Title I Choice Arrangements 

Describe the procedures used for the other choice options identified in Q. 1 above {address the following 
questions for each type of option separately): 

3.1 How are eligible students defined? 

3.2 How are students/parents notified of their choices? [Be sure to cover all the communication modes — 
e.g., see Q.5 in the School Survey instrument.) 

3.3 What criteria are used in selecting students to exercise their choice? 

[Be sure to cover all criteria — e.g., see Q3.3 in the VPSC field 
instrument.) 

Interview with the Comparison Site, 2006-07 
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3.4 What proportion of eligible students were able to exercise their choice? 

{Also determine the proportion receiving their 1st, 2nd, etc. choices; 
obtain data for the past 2-3 years, if possible. ] 

3.5 How have transportation services been affected, if at all, by the choice 

arrangement? {Be sure to cover ail facets of these services — e.g., see Q.6 in the 
VPSC instrument. ] 

3.6 Are other choice options likely to be implemented in the future? 



4. Trends Associated with Choice Options 

Determine whether the district maintains records of individual students who have transferred or of the 
performance of sending and receiving schools. The following questions may be addressed once, even if the 
district has more than one choice arrangement. 



4. 1 Are there any data showing trends in academic performance by the students who have exercised their 
choice? {Probe for data for the past 2-3 years, if possible. ] 

4.2 Are there any data showing trends in academic performance by the schools involved in the choice 
arrangement? {Probe for data for the past 2-3 years, if possible.^ 

4.3 What are the district’s policies, views, and preferences regarding school choice options? 

4.4 What interest has been expressed by parents, communities, or students regarding school choice 
options at this district? 



5. Contextual Conditions 

5. 1 Have there been any notable changes by the surrounding districts that might be relevant to either the 
choice options or their outcomes at this district? 

5.2 Have there been any changes in the community (e.g., residential relocation patterns) that might be 
relevant to either the choice options or their outcomes at this district? 

5.3 How has the district’s “share” of the student population changed, relative to that of private schools in 
the area? {Obtain annual data by grade level, if possible^ 

5.4 Is the district implementing other approaches to school improvement as an alternative to choice? 
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Exhibit C-1 

State Education Agencies Used VPSC Funds to 
Support Widely Different Choice Priorities 

State departments of education received three of the 13 VPSC awards. 

One did not use the VPSC funds to support any particuiar choice options directiy 
but instead used the funds to provide technicai assistance to its districts, to pian and 
impiement their choice options. The technicai assistance appeared to be an 
appropriate priority, given the state's historicaiiy wide array of choice options on top of 
options impiemented by individuai districts. 

A second state used the funds to support an urban-suburban interdistrict choice 
arrangement. Such an arrangement fuifilied the state’s commitment, stemming from 
the settiement of a raciai discrimination iawsuit, to provide opportunities for iow- 
income students in the state’s iargest urban district to transfer to suburban schoois. 

The third state initiated a new option whereby students did not need to attend 
their reguiar schoois but received Web-based curricuia and instruction at off-campus 
iocations. The state aiready had experienced some iegisiative controversy over 
supporting off-campus schooiing, and the new option provided an opportunity to 
strengthen such schooiing as weli as to test it by permitting students to take the 
annuai state assessments. 



Exhibit C-2 

VPSC Initiatives Coincided With 
Existing, Districtwide Choice Options 

Four of the 13 VPSC sites used the VPSC funds to enhance preexisting choice 
options whereby students couid choose to attend any schooi in the district. 

At three districts, the VPSC funds heiped the districts to strengthen the academic 
programs at under-enroiied schoois, making them more attractive, as weli as to 
formalize the selection and assignment procedures in exercising choice options. The 
fourth district already had been actively planning its new districtwide option, and the 
VPSC funds helped to support the implementation of the option — e.g., the kick-off 
marketing campaign to solicit choice applications, and the development of “attractor” 
programs and associated professional development at selected schools. 

At none of the four sites was it possible to disentangle the influence of VPSC 
funds from districts’ original designs and other sources of funding support. 
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Exhibit C-4 

VPSC Initiatives Could Be Separate From or 
Coincide With Title I Choice 



Two VPSC sites defined their VPSC initiatives to coincide directiy with their 
fuifiilment of Titie I choice requirements, defining sending schoois as Titie i schoois 
“identified for improvement” and receiving schoois as higher-performing schoois. At a 
third site, the specific set of iow-performing VPSC sending schoois inciuded but were 
not limited to the site's Titie I schoois. The remainder of the VPSC sites defined and 
administered their VPSC initiatives totaiiy apart from their Titie I choice options. 
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Exhibit C-5 



Five Years of VPSC Participants 



VPSC 

Site 


Data 
for 4 
Years 


2006- 

07 


2005- 

06 


Eligible 

2004- 

OS 


2003- 

04 


2002- 

OS 


2006- 

07 


2005- 

06 


Applying 

2004- 

OS 


2003- 

04 


2002- 

OS 


2006- 

07 


2005- 

06 


Enrollin 

2004- 

OS 


3 

2003- 

04 


2002- 

OS 


A 




n.a. 


290,142 


318,231 


279,800 


206,700 


n.a. 


214 


198 


774 


717 


n.a. 


214 


198 


450 


450 


B 


V 


2,594 


670 


0 


0 


0 


n.a. 


n.a. 


0 


0 


0 


266 


196 


0 


0 


0 


C* 


V 


311,481 


320,331 


300,000 


332,055 


0 


n.a. 


n.a. 


4,831 


4,487 


0 


11,371 


13,068 


878 


1,076 


0 


D 


V 


10,292 


10,452 


10,452 


6,775 


6,425 


161 


170 


608 


545 


37 


161 


170 


111 


55 


37 


E 




n.a. 


n.a. 


n.a. 


F 


V 


1,795 


1,682 


1,842 


720 


0 


1,795 


1,682 


1,842 


720 


0 


1,795 


1,682 


1,842 


720 


0 


G 


V 


163,159 


46,838 


28,349 


0 


0 


2,042 


1,620 


9,924 


0 


0 


1,969 


1,494 


4,270 


0 


0 


H 


V 


150,000 


134,878 


108,845 


45,971 


0 


2,312 


n.a. 


3,884 


162 


0 


901 


3,039 


2,755 


162 


0 




V 


25,333 


27,266 


29,531 


29,000 


0 


827 


674 


1,037 


3,417 


0 


827 


674 


1,037 


759 


0 


J 




n.a. 


8,235 


4,073 


3,941 


n.a. 


n.a. 


n.a. 


167 


126 


19 


n.a. 


121 


48 


31 


19 




V 


8,444 


4,970 


4,849 


1,725 


4,905 


1,205 


446 


741 


190 


501 


1,205 


446 


741 


191 


501 


L 


V 


45,970 


45,727 


51,000 


48,900 


0 


4,818 


5,249 


5,007 


4,936 


0 


3,144 


3,783 


4,249 


3,844 


0 


M 


V 


4,863 


4,993 


5,224 


6,500 


1,660 


76 


114 


149 


364 


117 


35 


34 


34 


157 


80 


Total 


723,931 


896,184 


862,396 


755,387 


219,690 


13,236 


10,169 


28,388 


15,721 


1,391 


21,674 


24,921 


16,163 


7,445 


1,087 


Total Number of 
Sites Reporting 


10 


12 


11 


10 


4 


8 


8 


11 


10 


5 


10 


12 


11 


10 


5 


Mean for the 10 
sites with 4 years 
of enrollment data 


72,393 


59,781 


54,009 


47,165 


n.a. 


n.a. 


2,167 


2,459 


1,592 


696 


n.a. 


Percent of eligible 
students enrolled 
at the 10 sites 




3.0 


4.1 


2.9 


1.5 


n.a. 



Exhibit Reads: Site A for 2005-06, reported 290,142 eiigibie, 214 appiying, and 214 enroiiing VPSC participants. 

*For 2005-06 and 2006-07, the site may have eounted transfers from earlier years along with first-time enrollees. 

**The number of eligible students reflects the site’s students, K-8, who are eligible for the Free and Reduced-Price Lunch Program. The number is an 
underestimate: suburban students who have the option to move to the site’s magnets are not included in the eligibility count. 

***The 2002-03 applicant number (190) and enrollee number (191) are as reported by the site. The site’s staff explained that an enrollee was placed without a 
formal application. 

n.a.= data were not available for that year. 

Sources: Grant Performance Reports, supplemented by site visit databases. 



Exhibit C-6 

Choice Initiatives Limited to Zones Within a District 
Nevertheless Involved a Broad Array of Sending Schools 

Two large urban VPSC sites designed zonal initiatives, giving special attention to 
schools in particular zones in their district. One site eventually scaled up to cover ten 
zones and 65 schools; the other had two zones totaling 106 schools, but VPSC funds 
focused on 23 of these 106 schools. The sites chose the zones because the schools 
in them had tended to be under-enrolled and underperforming. 

In theory, the restrictions to these zones, in lieu of districtwide coverage, would 
have classified these initiatives as having targeted selected schools. However, 
although most transferring students made within-zone transfers, at both sites students 
in the entire district were eligible to request transfers into the schools in these zones. 
Thus, the potential sending schools actually represented a broad array of schools, 
leading to the classification of these initiatives as “broad” and not “targeted.” 



Exhibit C-7 

Eligible Sending Schools Were Restricted 
but Not Necessarily to Low-Performing Schools 

Five VPSC sites limited the eligibility for their choice initiatives to students 
attending specific sending schools, and not all of the schools in their districts. 
However, the designations at two of the sites were based on other factors than school 
performance: at one site that had a statewide initiative, the sending schools were 
limited to those located in the districts that had agreed to participate in the initiative, 
not all of the districts in the state; at the other site that started a charter school 
initiative, the charters were mainly designed to attract students previously attending 
specific schools in the school district, but the schools did not have to be low- 
performing. 

In contrast, the three VPSC sites that limited eligibility to low-performing schools 
did choose those schools because of their performance on state assessments. 



Exhibit C-8 

Capacity-building Occurred at Receiving Schools 
That Already Were (Presumably) Higher-performing Schools 

Several of the VPSC sites enhanced academic programs at receiving schools 
even though these schools had already been designated as higher-performing 
schools. At one site, the academic changes and an infusion of computer technology 
benefited the students already enrolled at the receiving schools. At another site, a 
school previously designated as an acceptable receiving school, became “identified for 
improvement” midway through the VPSC initiative. At a third site, surveys showed 
that parents were not particularly swayed by “attractor” programs at receiving schools. 
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Exhibit C-9 

Local Sites May Need to Focus on a 
“Tapestry” of Choice Options, 

Rather Than on Any Single Initiative 

One of the two sites that had put into piace the fourth type of choice 
arrangement — a “mixture” of the first three arrangements (see section 3.2) — previousiy 
only had one choice option serving a multidistrict region. In hindsight, the “mixture” 
may have been the site’s way of responding to the diversity of local needs by trying to 
create a tapestry of choice eptions rather than focusing on a single type of initiative. 



Exhibit C-10 

“Attractor” Programs at Receiving Schools 
May Not Have Been Needed to Attract Incoming Students 

The VPSC sites frequently put new education programs into place at the receiving 
schools. One metive for such programs was that they would more readily attract 
students frem other schools, either by diversifying or intensifying the curriculum to 
match the incoming students' interests. 

One VPSC site invested considerable effort in defining and implementing a set of 
what it called “attractor” programs at a series of receiving schools. Two years later, 
the site's parent survey revealed that parents were attracted to the schools by their 
everall reputations, for academic excellence or even for school safety, but the parents’ 
preferences were not influenced by the presence of the attractor programs. The 
VPSC site subsequently reduced its emphasis on these programs. 



Exhibit C-II 

Explicit Efforts May Be Needed to Maintain the 
Performance of Receiving Schools 

One site’s choice initiative had predesignated sending schools that had been 
deemed low-performing according to NCLB standards. Students could transfer to 
other schools in the district, but the site also used VPSC funds to strengthen the 
academic programs at four specific receiving scheols that were lecated near the 
sending schools. The four schools also had students with backgrounds similar to 
those at the sending schools. Two years after VPSC enrollment had started, two of 
the feur receiving schools fell into the NCLB category of “identified for improvement.” 
Whether (as in the eyes of the VPSC initiative’s staff) students transferring from the 
low-performing schools exacerbated the receiving schools’ performance, explicit 
precautionary steps are needed to avert such eutcomes in the implementation of 
choice initiatives. 
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